This project was built by a team of four persons including Azizha Zeinita
Japan Exchange Group, Inc. (JPX) is a holding company operating one of the largest stock exchanges in the world, which one of it’s significant stocks is Tokyo Gas stock. This project will compare several models against real future returns& of Tokyo Gas stock after the training phase is complete. Other external datasets that are closely related to stock price, such as Japan GDP, Unemployment Rate, Oil Price, etc., will also involved in different combinations to find the best output.
Datasets
Stock Data
Tokyo Gas Stock (JPX)
Source:
External Data
Nikkei 225 index
Sources:
Time series models using ARIMA, VAR, PROPHET, Long Short-Term Memory (LSTM) with evaluation metrics MAE, MSE, MAPE, sMAPE, MASE
pip install sktime
Requirement already satisfied: sktime in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (0.11.2) Requirement already satisfied: scikit-learn>=0.24.0 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from sktime) (0.24.2) Requirement already satisfied: scipy<1.8.0 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from sktime) (1.7.1) Requirement already satisfied: numba>=0.53 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from sktime) (0.55.1) Requirement already satisfied: pandas<1.5.0,>=1.1.0 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from sktime) (1.3.4) Requirement already satisfied: statsmodels>=0.12.1 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from sktime) (0.12.2) Requirement already satisfied: deprecated>=1.2.13 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from sktime) (1.2.13) Requirement already satisfied: numpy<1.22,>=1.21.0 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from sktime) (1.21.6) Requirement already satisfied: wrapt<2,>=1.10 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from deprecated>=1.2.13->sktime) (1.12.1) Requirement already satisfied: setuptools in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from numba>=0.53->sktime) (58.0.4) Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from numba>=0.53->sktime) (0.38.0) Requirement already satisfied: python-dateutil>=2.7.3 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pandas<1.5.0,>=1.1.0->sktime) (2.8.2) Requirement already satisfied: pytz>=2017.3 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pandas<1.5.0,>=1.1.0->sktime) (2021.3) Requirement already satisfied: six>=1.5 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas<1.5.0,>=1.1.0->sktime) (1.16.0) Requirement already satisfied: joblib>=0.11 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from scikit-learn>=0.24.0->sktime) (1.1.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from scikit-learn>=0.24.0->sktime) (2.2.0) Requirement already satisfied: patsy>=0.5 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from statsmodels>=0.12.1->sktime) (0.5.2) Note: you may need to restart the kernel to use updated packages.
pip install pmdarima
Requirement already satisfied: pmdarima in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (1.8.5) Requirement already satisfied: statsmodels!=0.12.0,>=0.11 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pmdarima) (0.12.2) Requirement already satisfied: scikit-learn>=0.22 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pmdarima) (0.24.2) Requirement already satisfied: joblib>=0.11 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pmdarima) (1.1.0) Requirement already satisfied: Cython!=0.29.18,>=0.29 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pmdarima) (0.29.24) Requirement already satisfied: setuptools!=50.0.0,>=38.6.0 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pmdarima) (58.0.4) Requirement already satisfied: numpy>=1.19.3 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pmdarima) (1.21.6) Requirement already satisfied: pandas>=0.19 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pmdarima) (1.3.4) Requirement already satisfied: scipy>=1.3.2 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pmdarima) (1.7.1) Requirement already satisfied: urllib3 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pmdarima) (1.26.7) Requirement already satisfied: python-dateutil>=2.7.3 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pandas>=0.19->pmdarima) (2.8.2) Requirement already satisfied: pytz>=2017.3 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from pandas>=0.19->pmdarima) (2021.3) Requirement already satisfied: six>=1.5 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas>=0.19->pmdarima) (1.16.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from scikit-learn>=0.22->pmdarima) (2.2.0) Requirement already satisfied: patsy>=0.5 in /Users/azizhazeinita/opt/anaconda3/lib/python3.9/site-packages (from statsmodels!=0.12.0,>=0.11->pmdarima) (0.5.2) Note: you may need to restart the kernel to use updated packages.
# Basic libraries
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
import datetime as dt
import matplotlib.pyplot as plt
import seaborn as sns
import os
from scipy import stats
from sktime.forecasting.all import temporal_train_test_split
from tensorflow import keras
import tensorflow as tf
import statsmodels.api as sm
from scipy.special import inv_boxcox
from sktime.performance_metrics.forecasting import mean_absolute_error
from sktime.performance_metrics.forecasting import mean_squared_error
from sktime.performance_metrics.forecasting import mean_absolute_percentage_error
from pmdarima.arima import auto_arima
import warnings
warnings.filterwarnings('ignore')
# Function for simple EDA
def df_overview(df):
dtype_df = pd.DataFrame(df.dtypes, columns=["type"])
numNA_df = pd.DataFrame(df.isnull().sum(), columns=["NAs"])
nunique_df = pd.DataFrame(df.nunique(), columns=["unique"])
describe_df = df.describe(include="all").T[["count", "min", "max", "mean"]]
overview_df = pd.concat([dtype_df, numNA_df, nunique_df, describe_df], axis=1)
overview_df.fillna(value="-", inplace=True)
overview_df["count"] = overview_df["count"].astype(int)
overview_df["(%)"] = round((overview_df["NAs"]/df.shape[0])*100, 1)
overview_df.loc[overview_df["NAs"] == 0, "(%)"] = "-"
overview_df = overview_df.reindex(columns=["type", "count", "NAs", "(%)", "unique", "min", "max", "mean"])
display(overview_df)
# Libraries for ACF & PACF
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
# Libraries for ADF & KPSS
from statsmodels.tsa.stattools import kpss, adfuller
# Library for Ljung-Box test
from statsmodels.stats.diagnostic import acorr_ljungbox
# Function to run both ADF and KPSS tests
def adf_kpss(timeseries, alpha=0.05):
# ADF test
adf_test = adfuller(timeseries, autolag="AIC")
adf_result = pd.Series(adf_test[0:3],
index=["Test Statistic", "p-value", "Lags Used"])
for key, value in adf_test[4].items():
adf_result["Critical Value (%s)"%key] = value
# KPSS test with regression="c"
kpss_test_c = kpss(timeseries, regression="c")
kpss_result_c = pd.Series(kpss_test_c[0:3],
index=["Test Statistic", "p-value", "Lags Used"])
for key, value in kpss_test_c[3].items():
kpss_result_c["Critical Value (%s)"%key] = value
# KPSS test with regression="ct"
kpss_test_ct = kpss(timeseries, regression="ct")
kpss_result_ct = pd.Series(kpss_test_ct[0:3],
index=["Test Statistic", "p-value", "Lags Used"])
for key, value in kpss_test_ct[3].items():
kpss_result_ct["Critical Value (%s)"%key] = value
# Result table
result = pd.concat([adf_result, kpss_result_c, kpss_result_ct], axis=1).dropna()
result.columns = ["ADF", "KPSS('c')", "KPSS('ct')"]
# Determine whether timeseries is stationary or not from each test
adf_pval = result.loc["p-value", "ADF"]
adf_bool = adf_pval <= alpha
print(f"stationarity from ADF test: {adf_bool}")
kpss_c_pval = result.loc["p-value", "KPSS('c')"]
kpss_c_bool = kpss_c_pval >= alpha
print(f"stationarity from KPSS test('c'): {kpss_c_bool}")
kpss_ct_pval = result.loc["p-value", "KPSS('ct')"]
kpss_ct_bool = kpss_ct_pval >= alpha
print(f"stationarity from KPSS test('ct'): {kpss_ct_bool}\n")
# Show the result table
display(result)
# Function to show lineplot, histogram, and ADF/KPSS result for series
def create_plot_adf_kpss(timeseries, title, alpha=0.05):
fig, axes = plt.subplots(1,2, figsize=(16,5))
axes[0] = timeseries.plot(ax=axes[0])
axes[0].set_title(title)
axes[0].grid()
axes[1].hist(timeseries)
axes[1].set_title(f"Histogram of {title}")
plt.show()
print(f"\nADF/KPSS tests for {title} (alpha={alpha}):")
adf_kpss(timeseries, alpha)
# acf, pacf Import
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
# Function to plot ACF/PACF and PACF for timeseries
def plot_acf_pacf(timeseries, title, lags=50, auto_ylims=True):
fig, axes = plt.subplots(2,1, figsize=(16,10))
plot_acf(timeseries, lags=lags, title=f"ACF for {title}", ax=axes[0], zero=False, auto_ylims=auto_ylims)
axes[0].grid()
plot_pacf(timeseries, lags=lags, title=f"PACF for {title}", ax=axes[1], zero=False, auto_ylims=auto_ylims)
axes[1].grid()
plt.show()
# Libraries for metrics
from sktime.performance_metrics.forecasting import MeanAbsolutePercentageError
from sktime.performance_metrics.forecasting import MeanAbsoluteError
from sktime.performance_metrics.forecasting import MeanSquaredError
def calc_metrics(y_true, y_pred, round_value=4, print_result=True, return_dict=False):
# Create dict
metrics_dict = dict()
# MAE
mae = MeanAbsoluteError()
metrics_dict["MAE"] = mae(y_true, y_pred)
# MSE
mse = MeanSquaredError(square_root=False)
metrics_dict["MSE"] = mse(y_true, y_pred)
# MAPE
mape = MeanAbsolutePercentageError(symmetric=False)
metrics_dict["MAPE"] = mape(y_true, y_pred)
# SMAPE
smape = MeanAbsolutePercentageError(symmetric=True)
metrics_dict["sMAPE"] = smape(y_true, y_pred)
# MASE(for monthly seasonal timeseries)
naive_foacast_error = y_true.diff(12).mean()
metrics_dict["MASE"] = mae(y_true, y_pred)/naive_foacast_error
# Round values
for key, value in metrics_dict.items():
metrics_dict[key] = round(value, round_value)
# Print result or return metrics_dict
if print_result:
print("Test Error Metrics")
for key, value in metrics_dict.items():
print(f"{key}: {value}")
if return_dict:
return metrics_dict
def make_plot(train_y, test_y, pred_y, title):
fig, ax = plt.subplots(figsize=(16,5))
train_y.plot(ax=ax, label="Training")
test_y.plot(ax=ax, label="Test", color="black")
pred_y.plot(ax=ax, label=title, linestyle="dashed")
plt.legend()
plt.show()
PATH = "/Users/azizhazeinita/Project/Tokyo Gas/"
df_listings = pd.read_csv(PATH + 'stock_list.csv')
df_stock_price = pd.read_csv(PATH + 'stock_prices.csv')
# Apply => No NAs, dtype:int or float
print("df_stock_price")
df_overview(df_stock_price)
print("\ndf_listings")
df_overview(df_listings)
df_stock_price
| type | count | NAs | (%) | unique | min | max | mean | |
|---|---|---|---|---|---|---|---|---|
| RowId | object | 2332531 | 0 | - | 2332531 | - | - | - |
| Date | object | 2332531 | 0 | - | 1202 | - | - | - |
| SecuritiesCode | int64 | 2332531 | 0 | - | 2000 | 1301.0 | 9997.0 | 5894.835185 |
| Open | float64 | 2324923 | 7608 | 0.3 | 23067 | 14.0 | 109950.0 | 2594.510783 |
| High | float64 | 2324923 | 7608 | 0.3 | 23960 | 15.0 | 110500.0 | 2626.539682 |
| Low | float64 | 2324923 | 7608 | 0.3 | 23904 | 13.0 | 107200.0 | 2561.227039 |
| Close | float64 | 2324923 | 7608 | 0.3 | 24046 | 14.0 | 109550.0 | 2594.023446 |
| Volume | int64 | 2332531 | 0 | - | 89006 | 0.0 | 643654000.0 | 691936.556556 |
| AdjustmentFactor | float64 | 2332531 | 0 | - | 19 | 0.1 | 20.0 | 1.000508 |
| ExpectedDividend | float64 | 18865 | 2313666 | 99.2 | 446 | 0.0 | 1070.0 | 22.01773 |
| SupervisionFlag | bool | 2332531 | 0 | - | 2 | - | - | - |
| Target | float64 | 2332293 | 238 | 0.0 | 354507 | -0.578541 | 1.119512 | 0.000445 |
df_listings
| type | count | NAs | (%) | unique | min | max | mean | |
|---|---|---|---|---|---|---|---|---|
| SecuritiesCode | int64 | 4417 | 0 | - | 4417 | 1301.0 | 25935.0 | 5547.191306 |
| EffectiveDate | int64 | 4417 | 0 | - | 10 | 20000101.0 | 20220225.0 | 20197579.726738 |
| Name | object | 4417 | 0 | - | 4411 | - | - | - |
| Section/Products | object | 4417 | 0 | - | 11 | - | - | - |
| NewMarketSegment | object | 3772 | 645 | 14.6 | 6 | - | - | - |
| 33SectorCode | object | 4417 | 0 | - | 34 | - | - | - |
| 33SectorName | object | 4417 | 0 | - | 34 | - | - | - |
| 17SectorCode | object | 4417 | 0 | - | 18 | - | - | - |
| 17SectorName | object | 4417 | 0 | - | 34 | - | - | - |
| NewIndexSeriesSizeCode | object | 4417 | 0 | - | 6 | - | - | - |
| NewIndexSeriesSize | object | 4417 | 0 | - | 6 | - | - | - |
| TradeDate | float64 | 4121 | 296 | 6.7 | 10 | 20211216.0 | 20211230.0 | 20211229.954137 |
| Close | float64 | 4121 | 296 | 6.7 | 2566 | 1.8 | 711000.0 | 6376.7347 |
| IssuedShares | float64 | 4121 | 296 | 6.7 | 4011 | 2242.0 | 16314987460.0 | 89922667.041686 |
| MarketCapitalization | float64 | 4121 | 296 | 6.7 | 4121 | 38786600.0 | 52192239810720.0 | 218440012227.214966 |
| Universe0 | bool | 4417 | 0 | - | 2 | - | - | - |
jp_gdp.csvjp_unemp.csvoil.csvusdjpy.csvjp_interest_rate_long.csvjp_n225.csv# Function to load the data and create series
def load_(path, f_name, col_name):
# Load dataframe
df = pd.read_csv(os.path.join(path, f_name))
df["Date"] = pd.to_datetime(df["Date"])
df = df.set_index(df["Date"])
df.columns = ["Date", col_name]
# Extract needed external data (other than "Date" column) as series
series = df[col_name]
# Return both df and series
return df, series
# Path
PATH_ = "/Users/azizhazeinita/Project/Tokyo Gas/external_data/"
# GDP
gdp, gdp_s = load_(PATH_, "jp_gdp.csv", "jp_gdp")
# Unemployment rate
unemp, unemp_s = load_(PATH_, "jp_unemp.csv", "jp_unemp")
# Oil price
oil, oil_s = load_(PATH_, "oil.csv", "oil")
# JPY/USD
usdjpy, usdjpy_s = load_(PATH_, "usdjpy.csv", "usdjpy")
# Interest Rate (long-term)
interest, interest_s = load_(PATH_, "jp_interest_rate_long.csv", "jp_i")
# Nikkei 225 index
n225, n225_s = load_(PATH_, "jp_n225.csv", "n225")
temp = df_listings.groupby('33SectorName').count().sort_values('SecuritiesCode', ascending=False)
temp.head(10)
| SecuritiesCode | EffectiveDate | Name | Section/Products | NewMarketSegment | 33SectorCode | 17SectorCode | 17SectorName | NewIndexSeriesSizeCode | NewIndexSeriesSize | TradeDate | Close | IssuedShares | MarketCapitalization | Universe0 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 33SectorName | |||||||||||||||
| Information & Communication | 551 | 551 | 551 | 551 | 538 | 551 | 551 | 551 | 551 | 551 | 539 | 539 | 539 | 539 | 551 |
| Services | 521 | 521 | 521 | 521 | 502 | 521 | 521 | 521 | 521 | 521 | 506 | 506 | 506 | 506 | 521 |
| - | 500 | 500 | 500 | 500 | 10 | 500 | 500 | 500 | 500 | 500 | 345 | 345 | 345 | 345 | 500 |
| Retail Trade | 359 | 359 | 359 | 359 | 337 | 359 | 359 | 359 | 359 | 359 | 339 | 339 | 339 | 339 | 359 |
| Wholesale Trade | 320 | 320 | 320 | 320 | 311 | 320 | 320 | 320 | 320 | 320 | 311 | 311 | 311 | 311 | 320 |
| Electric Appliances | 255 | 255 | 255 | 255 | 243 | 255 | 255 | 255 | 255 | 255 | 243 | 243 | 243 | 243 | 255 |
| Machinery | 230 | 230 | 230 | 230 | 227 | 230 | 230 | 230 | 230 | 230 | 228 | 228 | 228 | 228 | 230 |
| Chemicals | 218 | 218 | 218 | 218 | 214 | 218 | 218 | 218 | 218 | 218 | 214 | 214 | 214 | 214 | 218 |
| Construction | 170 | 170 | 170 | 170 | 157 | 170 | 170 | 170 | 170 | 170 | 157 | 157 | 157 | 157 | 170 |
| Real Estate | 143 | 143 | 143 | 143 | 137 | 143 | 143 | 143 | 143 | 143 | 137 | 137 | 137 | 137 | 143 |
# The number of companies in each industry
temp = df_listings.groupby('33SectorName').count().sort_values('SecuritiesCode', ascending=False)
plt.figure(figsize=(20,15))
sns.barplot(y = temp.index, x = temp['Name'], orient="h",
palette = sns.color_palette("GnBu_r"))
plt.ylabel('Sector Name', fontsize = 14)
plt.xlabel('Count', fontsize = 10)
plt.title('Count of Industry')
plt.tick_params(axis='both', which='major', labelsize=20)
plt.show()
#The top 10 industry intern of nunmber
temp2 = df_listings.groupby('33SectorName').count().sort_values('SecuritiesCode', ascending=False).head(10)
plt.figure(figsize=(20,15))
sns.barplot(y = temp2.index, x = temp2['Name'], orient="h",
palette = sns.color_palette("GnBu_r"))
plt.ylabel('Sector Name', fontsize = 20)
plt.xlabel('Count', fontsize = 20)
plt.title('Top 10 Industry', fontsize = 20)
plt.tick_params(axis='both', which='major', labelsize=20)
plt.show()
def create_df_energy(df_raw=df_listings):
# Elec & Gas
df_gas = df_raw[(df_raw["33SectorName"] == "Electric Power and Gas")]
df_gas.loc[df_gas["Name"].str.contains("Power"), "Category"] = "Electric Utility"
df_gas.loc[df_gas["Name"].str.contains("GAS"), "Category"] = "Gas Utility"
df_gas.loc[df_gas["Name"].str.contains("METAWATER"), "Category"] = "Other"
df_gas["Category"] = df_gas["Category"].fillna("Other Electricity Service")
# Oil & Coal
df_oil = df_raw[(df_raw["33SectorName"] == "Oil and Coal Products")]
oil_company = ["ENEOS Holdings,Inc.", "Idemitsu Kosan Co.,Ltd.", "COSMO ENERGY HOLDINGS COMPANY,LIMITED",
"TonenGeneral Sekiyu K.K.", "Fuji Oil Company,Ltd.", "TOA OIL COMPANY,LIMITED"]
df_oil.loc[df_oil["Name"].isin(oil_company), "Category"] = "Oil"
df_oil.loc[df_oil["Name"]=="VITEC HOLDINGS CO.,LTD.", "Category"] = "Other"
df_oil["Category"] = df_oil["Category"].fillna("Oil Products Manufacturer")
# Merge
df_energy = pd.concat([df_gas, df_oil], axis=0)
return df_energy
df_energy = create_df_energy()
df_energy[["SecuritiesCode", "Name", "33SectorName", "Category", "Section/Products"]]
| SecuritiesCode | Name | 33SectorName | Category | Section/Products | |
|---|---|---|---|---|---|
| 4181 | 9501 | Tokyo Electric Power Company Holdings,Incorpor... | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4182 | 9502 | Chubu Electric Power Company,Incorporated | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4183 | 9503 | The Kansai Electric Power Company,Incorporated | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4184 | 9504 | The Chugoku Electric Power Company,Incorporated | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4185 | 9505 | Hokuriku Electric Power Company | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4186 | 9506 | Tohoku Electric Power Company,Incorporated | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4187 | 9507 | Shikoku Electric Power Company,Incorporated | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4188 | 9508 | Kyushu Electric Power Company,Incorporated | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4189 | 9509 | Hokkaido Electric Power Company,Incorporated | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4190 | 9511 | The Okinawa Electric Power Company,Incorporated | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4191 | 9513 | Electric Power Development Co.,Ltd. | Electric Power and Gas | Electric Utility | First Section (Domestic) |
| 4192 | 9514 | EF-ON INC. | Electric Power and Gas | Other Electricity Service | First Section (Domestic) |
| 4193 | 9517 | eREX Co.,Ltd. | Electric Power and Gas | Other Electricity Service | First Section (Domestic) |
| 4194 | 9519 | RENOVA,Inc. | Electric Power and Gas | Other Electricity Service | First Section (Domestic) |
| 4195 | 9522 | Renewable Japan Co.,Ltd. | Electric Power and Gas | Other Electricity Service | Mothers (Domestic) |
| 4196 | 9531 | TOKYO GAS CO.,LTD. | Electric Power and Gas | Gas Utility | First Section (Domestic) |
| 4197 | 9532 | OSAKA GAS CO.,LTD. | Electric Power and Gas | Gas Utility | First Section (Domestic) |
| 4198 | 9533 | TOHO GAS CO.,LTD. | Electric Power and Gas | Gas Utility | First Section (Domestic) |
| 4199 | 9534 | HOKKAIDO GAS CO.,LTD. | Electric Power and Gas | Gas Utility | First Section (Domestic) |
| 4200 | 9535 | HIROSHIMA GAS CO.,LTD. | Electric Power and Gas | Gas Utility | First Section (Domestic) |
| 4201 | 9536 | SAIBU GAS HOLDINGS CO.,LTD. | Electric Power and Gas | Gas Utility | First Section (Domestic) |
| 4202 | 9537 | HOKURIKU GAS CO.,LTD. | Electric Power and Gas | Gas Utility | Second Section(Domestic) |
| 4203 | 9539 | KEIYO GAS CO.,LTD. | Electric Power and Gas | Gas Utility | Second Section(Domestic) |
| 4204 | 9543 | SHIZUOKA GAS CO.,LTD. | Electric Power and Gas | Gas Utility | First Section (Domestic) |
| 4205 | 9551 | METAWATER Co.,Ltd. | Electric Power and Gas | Other | First Section (Domestic) |
| 1044 | 3315 | NIPPON COKE & ENGINEERING COMPANY,LIMITED | Oil and Coal Products | Oil Products Manufacturer | First Section (Domestic) |
| 2060 | 5002 | VITEC HOLDINGS CO.,LTD. | Oil and Coal Products | Other | First Section (Domestic) |
| 2061 | 5008 | TOA OIL COMPANY,LIMITED | Oil and Coal Products | Oil | Second Section(Domestic) |
| 2063 | 5010 | NIPPON SEIRO CO.,LTD. | Oil and Coal Products | Oil Products Manufacturer | Second Section(Domestic) |
| 2064 | 5011 | NICHIREKI CO.,LTD. | Oil and Coal Products | Oil Products Manufacturer | First Section (Domestic) |
| 2065 | 5012 | TonenGeneral Sekiyu K.K. | Oil and Coal Products | Oil | First Section (Domestic) |
| 2066 | 5013 | Yushiro Chemical Industry Co.,Ltd. | Oil and Coal Products | Oil Products Manufacturer | First Section (Domestic) |
| 2067 | 5015 | BP Castrol K.K. | Oil and Coal Products | Oil Products Manufacturer | First Section (Domestic) |
| 2068 | 5017 | Fuji Oil Company,Ltd. | Oil and Coal Products | Oil | First Section (Domestic) |
| 2069 | 5018 | MORESCO Corporation | Oil and Coal Products | Oil Products Manufacturer | First Section (Domestic) |
| 2070 | 5019 | Idemitsu Kosan Co.,Ltd. | Oil and Coal Products | Oil | First Section (Domestic) |
| 2071 | 5020 | ENEOS Holdings,Inc. | Oil and Coal Products | Oil | First Section (Domestic) |
| 2072 | 5021 | COSMO ENERGY HOLDINGS COMPANY,LIMITED | Oil and Coal Products | Oil | First Section (Domestic) |
df_energy.shape
(38, 17)
# Merge df_stock_price & df_energy
df = pd.merge(
left=df_stock_price,
right=df_energy[["SecuritiesCode", "Name", "33SectorName", "Category", "Section/Products"]],
on="SecuritiesCode", how="left"
).dropna(subset=["Name"])
df["Date"] = pd.to_datetime(df["Date"])
df = df.set_index(df["Date"], drop=True)
# Comapnies not included in stock_price.csv
df_energy[df_energy["Name"].isin(list(set(df_energy["Name"].unique()).difference(set(df["Name"].unique()))))]
| SecuritiesCode | EffectiveDate | Name | Section/Products | NewMarketSegment | 33SectorCode | 33SectorName | 17SectorCode | 17SectorName | NewIndexSeriesSizeCode | NewIndexSeriesSize | TradeDate | Close | IssuedShares | MarketCapitalization | Universe0 | Category | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 4192 | 9514 | 20211230 | EF-ON INC. | First Section (Domestic) | Prime Market | 4050 | Electric Power and Gas | 11 | ELECTRIC POWER & GAS | 7 | TOPIX Small 2 | 20211230.0 | 618.0 | 21636579.0 | 1.337141e+10 | False | Other Electricity Service |
| 4195 | 9522 | 20211230 | Renewable Japan Co.,Ltd. | Mothers (Domestic) | Growth Market | 4050 | Electric Power and Gas | 11 | ELECTRIC POWER & GAS | - | - | 20211230.0 | 1709.0 | 28420910.0 | 4.857134e+10 | False | Other Electricity Service |
| 2060 | 5002 | 20000101 | VITEC HOLDINGS CO.,LTD. | First Section (Domestic) | NaN | 3300 | Oil and Coal Products | 2 | ENERGY RESOURCES | 4 | TOPIX Mid400 | NaN | NaN | NaN | NaN | False | Other |
| 2063 | 5010 | 20211230 | NIPPON SEIRO CO.,LTD. | Second Section(Domestic) | Standard Market | 3300 | Oil and Coal Products | 2 | ENERGY RESOURCES | - | - | 20211230.0 | 191.0 | 22400000.0 | 4.278400e+09 | False | Oil Products Manufacturer |
| 2065 | 5012 | 20000101 | TonenGeneral Sekiyu K.K. | First Section (Domestic) | NaN | 3300 | Oil and Coal Products | 2 | ENERGY RESOURCES | 4 | TOPIX Mid400 | NaN | NaN | NaN | NaN | False | Oil |
| 2069 | 5018 | 20211230 | MORESCO Corporation | First Section (Domestic) | Prime Market | 3300 | Oil and Coal Products | 2 | ENERGY RESOURCES | 7 | TOPIX Small 2 | 20211230.0 | 1161.0 | 9696500.0 | 1.125764e+10 | False | Oil Products Manufacturer |
fig, ax = plt.subplots(figsize=(16,8))
for c in df.loc[df["Category"]=="Electric Utility", "Name"].unique():
df.loc[df["Name"]==c, "Close"].plot(ax=ax, label=c)
ax.legend(loc="upper right")
ax.set_title("Electric Utilities")
plt.show()
fig, ax = plt.subplots(figsize=(16,8))
for c in df.loc[df["Category"]=="Gas Utility", "Name"].unique():
df.loc[df["Name"]==c, "Close"].plot(ax=ax, label=c)
ax.legend(loc="upper left")
ax.set_title("Gas Utilities")
plt.show()
fig, ax = plt.subplots(figsize=(16,8))
for c in df.loc[df["Category"]=="Oil", "Name"].unique():
df.loc[df["Name"]==c, "Close"].plot(ax=ax, label=c)
ax.legend(loc="upper left")
ax.set_title("Oil Companies")
plt.show()
fig, ax = plt.subplots(figsize=(16,8))
for c in df.loc[df["Category"].isin(["Other Electricity Service", "Oil Products Manufacturer", "Other"]), "Name"].unique():
df.loc[df["Name"]==c, "Close"].plot(ax=ax, label=c)
ax.legend(loc="upper left")
ax.set_title("Other")
plt.show()
main5 = df.copy().loc[df["SecuritiesCode"].isin([9501, 9503, 9531, 9532, 5020]), ["Date", "SecuritiesCode", "Close", "Name", "Category"]]
main5.loc[main5["Name"]=="ENEOS Holdings,Inc.", "Name"] = "ENEOS"
main5.loc[main5["Name"]=="Tokyo Electric Power Company Holdings,Incorporated", "Name"] = "Tokyo Electric Power"
main5.loc[main5["Name"]=="The Kansai Electric Power Company,Incorporated", "Name"] = "Kansai Electric Power"
main5.loc[main5["Name"]=="TOKYO GAS CO.,LTD.", "Name"] = "Tokyo Gas"
main5.loc[main5["Name"]=="OSAKA GAS CO.,LTD.", "Name"] = "Osaka Gas"
fig, ax = plt.subplots(figsize=(16,8))
for c in main5["Name"].unique():
main5.loc[main5["Name"]==c, "Close"].plot(ax=ax, label=c)
ax.legend(loc="upper left")
ax.set_title("Main Utilities and Oil Companies", fontsize = 20)
plt.show()
# Tokyo Gas & Osaka Gas change unit of stocks from 9/27/2017 (5 shares to 1 share)
main5.loc[(main5["SecuritiesCode"].isin([9531, 9532])) & (main5.index.year==2017) & (main5.index.month==9) & (main5.index.day>20)].sort_values("SecuritiesCode")
| Date | SecuritiesCode | Close | Name | Category | |
|---|---|---|---|---|---|
| Date | |||||
| 2017-09-21 | 2017-09-21 | 9531 | 564.7 | Tokyo Gas | Gas Utility |
| 2017-09-22 | 2017-09-22 | 9531 | 567.7 | Tokyo Gas | Gas Utility |
| 2017-09-25 | 2017-09-25 | 9531 | 560.1 | Tokyo Gas | Gas Utility |
| 2017-09-26 | 2017-09-26 | 9531 | 564.4 | Tokyo Gas | Gas Utility |
| 2017-09-27 | 2017-09-27 | 9531 | 2763.0 | Tokyo Gas | Gas Utility |
| 2017-09-28 | 2017-09-28 | 9531 | 2844.0 | Tokyo Gas | Gas Utility |
| 2017-09-29 | 2017-09-29 | 9531 | 2758.0 | Tokyo Gas | Gas Utility |
| 2017-09-21 | 2017-09-21 | 9532 | 420.4 | Osaka Gas | Gas Utility |
| 2017-09-22 | 2017-09-22 | 9532 | 422.1 | Osaka Gas | Gas Utility |
| 2017-09-25 | 2017-09-25 | 9532 | 425.2 | Osaka Gas | Gas Utility |
| 2017-09-26 | 2017-09-26 | 9532 | 429.8 | Osaka Gas | Gas Utility |
| 2017-09-27 | 2017-09-27 | 9532 | 2094.5 | Osaka Gas | Gas Utility |
| 2017-09-28 | 2017-09-28 | 9532 | 2123.0 | Osaka Gas | Gas Utility |
| 2017-09-29 | 2017-09-29 | 9532 | 2092.0 | Osaka Gas | Gas Utility |
main5["Adjusted Close"] = main5["Close"]
main5.loc[(main5["SecuritiesCode"].isin([9531, 9532])) & (main5["Date"] < dt.datetime(2017,9,27)), "Adjusted Close"] = main5.loc[(main5["SecuritiesCode"].isin([9531, 9532])) & (main5["Date"] < dt.datetime(2017,9,27)), "Adjusted Close"]*5
fig, ax = plt.subplots(figsize=(16,8))
for c in main5["Name"].unique():
main5.loc[main5["Name"]==c, "Adjusted Close"].plot(ax=ax, label=c)
ax.legend(loc="upper right")
ax.set_title("Main Utilities and Oil Companies")
plt.show()
# See correlation
main5_corr = main5.copy()
main5_corr = main5_corr.pivot(index=["Date"], columns="Name", values="Adjusted Close")
sns.pairplot(main5_corr)
plt.show()
sns.heatmap(main5_corr.corr(), annot=True)
plt.show()
# Function to create plot to see seasonality
def check_seasonality(timeseries, title, ax=None):
df = pd.DataFrame(timeseries)
df.columns = ["price"]
df["year"] = df.index.year
df["dayofyear"] = df.index.dayofyear
if ax==None:
fig, ax = plt.subplots(figsize=(16,8))
sns.lineplot("dayofyear", "price", data=df, hue="year", ax=ax, legend=None)
ax.set_title(f"Seasonal Plot of {title}")
ax.set_xlabel("Day of Year")
ax.set_ylabel(title)
ax.grid()
plt.show()
else:
sns.lineplot("dayofyear", "price", data=df, hue="year", ax=ax, legend=None)
ax.set_title(f"Seasonal Plot of {title}")
ax.set_xlabel("Day of Year")
ax.set_ylabel(title)
ax.grid()
return ax
# Check seasonality for main 5 companies => Seem no seasonality
fig, axes = plt.subplots(3,2, figsize=(16,10), tight_layout=True)
axes_flat = [s for sublist in axes for s in sublist]
for i, name in enumerate(main5["Name"].unique()):
check_seasonality(main5.loc[main5["Name"]==name, "Adjusted Close"], name, ax=axes_flat[i])
plt.show()
fig, ax = plt.subplots(figsize=(16,8))
for c in main5["Name"].unique():
main5.loc[main5["Name"]==c, "Adjusted Close"].plot(ax=ax, label=c)
ax.set_title("Main Utilities and Oil Companies")
ax.plot(n225["n225"]/10, color="Black", label="Nikkei Stock Index (/10)")
ax.plot(usdjpy["usdjpy"]*10, color="grey", label="USDJPY (*10)")
ax.plot(oil["oil"]*50, color="yellow", label="WTI Oil Price (*50)")
ax.legend(loc="upper left")
ax.set_ylim(0,)
plt.show()
tokyo_gas = main5.loc[main5["Name"]=="Tokyo Gas", "Adjusted Close"]
tokyo_gas = tokyo_gas.dropna()
tokyo_gas.name = "Tokyo Gas"
# Check stationarity
create_plot_adf_kpss(tokyo_gas, "Tokyo_Gas_original")
ADF/KPSS tests for Tokyo_Gas_original (alpha=0.05):
stationarity from ADF test: False
stationarity from KPSS test('c'): False
stationarity from KPSS test('ct'): False
| ADF | KPSS('c') | KPSS('ct') | |
|---|---|---|---|
| Test Statistic | -1.201363 | 3.438455 | 0.722794 |
| p-value | 0.673009 | 0.010000 | 0.010000 |
| Lags Used | 9.000000 | 23.000000 | 23.000000 |
| Critical Value (1%) | -3.435852 | 0.739000 | 0.216000 |
| Critical Value (5%) | -2.863970 | 0.463000 | 0.146000 |
| Critical Value (10%) | -2.568064 | 0.347000 | 0.119000 |
# Take first difference
tokyo_gas_diff = tokyo_gas.diff().dropna()
# Check stationarity
create_plot_adf_kpss(tokyo_gas_diff, "Tokyo Gas (diff(1))")
ADF/KPSS tests for Tokyo Gas (diff(1)) (alpha=0.05):
stationarity from ADF test: True
stationarity from KPSS test('c'): True
stationarity from KPSS test('ct'): True
| ADF | KPSS('c') | KPSS('ct') | |
|---|---|---|---|
| Test Statistic | -1.315887e+01 | 0.089789 | 0.0155 |
| p-value | 1.319894e-24 | 0.100000 | 0.1000 |
| Lags Used | 8.000000e+00 | 23.000000 | 23.0000 |
| Critical Value (1%) | -3.435852e+00 | 0.739000 | 0.2160 |
| Critical Value (5%) | -2.863970e+00 | 0.463000 | 0.1460 |
| Critical Value (10%) | -2.568064e+00 | 0.347000 | 0.1190 |
plot_acf_pacf(tokyo_gas_diff, "tokyo_gas_dif(diff(1))", lags=50)
# Using data before 2020 as training
y_train = tokyo_gas[tokyo_gas.index.year!=2021]
y_test = tokyo_gas[tokyo_gas.index.year==2021]
arima_model = auto_arima(y_train, start_p=0, d=1, start_q=0, seasonal=False)
print(arima_model.summary())
SARIMAX Results
==============================================================================
Dep. Variable: y No. Observations: 975
Model: SARIMAX(4, 1, 1) Log Likelihood -4942.886
Date: Wed, 05 Oct 2022 AIC 9897.771
Time: 16:54:34 BIC 9927.060
Sample: 0 HQIC 9908.917
- 975
Covariance Type: opg
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
ar.L1 -0.7169 0.103 -6.935 0.000 -0.920 -0.514
ar.L2 -0.0480 0.029 -1.646 0.100 -0.105 0.009
ar.L3 -0.1299 0.034 -3.864 0.000 -0.196 -0.064
ar.L4 -0.1585 0.027 -5.860 0.000 -0.211 -0.105
ma.L1 0.7350 0.104 7.064 0.000 0.531 0.939
sigma2 1497.6393 47.041 31.837 0.000 1405.440 1589.839
===================================================================================
Ljung-Box (L1) (Q): 0.01 Jarque-Bera (JB): 378.52
Prob(Q): 0.92 Prob(JB): 0.00
Heteroskedasticity (H): 1.43 Skew: 0.07
Prob(H) (two-sided): 0.00 Kurtosis: 6.05
===================================================================================
Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
y_pred_arima110 = arima_model.predict(len(y_test))
y_pred_arima110 = pd.Series(y_pred_arima110)
y_pred_arima110.index = y_test.index
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_arima110.plot(ax=ax, label="ARIMA(1,1,0)", linestyle="dashed")
plt.legend()
plt.show()
arima110_metrics = calc_metrics(y_test, y_pred_arima110, return_dict=True)
Test Error Metrics MAE: 203.792 MSE: 58725.0992 MAPE: 0.0978 sMAPE: 0.0914 MASE: -8.7564
# Merge data and fill NAs with interpolation
tokyo_gas_m = pd.concat([tokyo_gas, oil_s, n225_s], axis=1)
tokyo_gas_m = tokyo_gas_m.dropna(subset=["Tokyo Gas"])
tokyo_gas_m["oil"] = tokyo_gas_m["oil"].interpolate()
tokyo_gas_m["n225"] = tokyo_gas_m["n225"].interpolate()
tokyo_gas_m
| Tokyo Gas | oil | n225 | |
|---|---|---|---|
| Date | |||
| 2017-01-04 | 2691.5 | 53.26 | 19594.160156 |
| 2017-01-05 | 2683.5 | 53.77 | 19520.689453 |
| 2017-01-06 | 2691.5 | 53.98 | 19454.330078 |
| 2017-01-10 | 2628.0 | 50.82 | 19301.439453 |
| 2017-01-11 | 2648.5 | 52.19 | 19364.669922 |
| ... | ... | ... | ... |
| 2021-11-29 | 1993.0 | 69.88 | 28283.919922 |
| 2021-11-30 | 1945.0 | 66.14 | 27821.759766 |
| 2021-12-01 | 1904.0 | 65.44 | 27935.619141 |
| 2021-12-02 | 1975.0 | 66.60 | 27753.369141 |
| 2021-12-03 | 1978.0 | 66.39 | 28029.570312 |
1201 rows × 3 columns
# Using data before 2020 as training
X_train_oil = tokyo_gas_m.loc[tokyo_gas_m.index.year!=2021, "oil"]
X_test_oil = tokyo_gas_m.loc[tokyo_gas_m.index.year==2021, "oil"]
arimax_oil = auto_arima(y_train, X=X_train_oil.values.reshape(-1, 1), start_p=0, d=1, start_q=0, seasonal=False)
print(arimax_oil.summary())
SARIMAX Results
==============================================================================
Dep. Variable: y No. Observations: 975
Model: SARIMAX(4, 1, 1) Log Likelihood -4942.464
Date: Wed, 05 Oct 2022 AIC 9898.927
Time: 16:55:29 BIC 9933.097
Sample: 0 HQIC 9911.931
- 975
Covariance Type: opg
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
x1 0.4327 0.776 0.558 0.577 -1.088 1.953
ar.L1 -0.7055 0.109 -6.459 0.000 -0.920 -0.491
ar.L2 -0.0480 0.029 -1.655 0.098 -0.105 0.009
ar.L3 -0.1269 0.034 -3.781 0.000 -0.193 -0.061
ar.L4 -0.1551 0.027 -5.706 0.000 -0.208 -0.102
ma.L1 0.7242 0.110 6.577 0.000 0.508 0.940
sigma2 1496.3368 47.401 31.568 0.000 1403.433 1589.241
===================================================================================
Ljung-Box (L1) (Q): 0.01 Jarque-Bera (JB): 392.25
Prob(Q): 0.92 Prob(JB): 0.00
Heteroskedasticity (H): 1.43 Skew: 0.08
Prob(H) (two-sided): 0.00 Kurtosis: 6.10
===================================================================================
Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
y_pred_arimax110_oil = arimax_oil.predict(len(y_test), X=X_test_oil.values.reshape(-1, 1))
y_pred_arimax110_oil = pd.Series(y_pred_arimax110_oil)
y_pred_arimax110_oil.index = y_test.index
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_arimax110_oil.plot(ax=ax, label="ARIMAX(1,1,0) with Oil Price", linestyle="dashed")
plt.legend()
plt.show()
arima110_oil_metrics = calc_metrics(y_test, y_pred_arimax110_oil, return_dict=True)
Test Error Metrics MAE: 211.5135 MSE: 63002.5703 MAPE: 0.1015 sMAPE: 0.0946 MASE: -9.0882
# Using data before 2020 as training
X_train_n225 = tokyo_gas_m.loc[tokyo_gas_m.index.year!=2021, "n225"]
X_test_n225 = tokyo_gas_m.loc[tokyo_gas_m.index.year==2021, "n225"]
arimax_n225 = auto_arima(y_train, X=X_train_n225.values.reshape(-1, 1), start_p=0, d=1, start_q=0, seasonal=False)
print(arimax_n225.summary())
SARIMAX Results
==============================================================================
Dep. Variable: y No. Observations: 975
Model: SARIMAX(0, 1, 0) Log Likelihood -4876.087
Date: Thu, 02 Jun 2022 AIC 9756.175
Time: 22:06:01 BIC 9765.938
Sample: 0 HQIC 9759.890
- 975
Covariance Type: opg
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
x1 0.0627 0.003 19.136 0.000 0.056 0.069
sigma2 1305.8502 37.170 35.132 0.000 1232.999 1378.701
===================================================================================
Ljung-Box (L1) (Q): 0.60 Jarque-Bera (JB): 383.36
Prob(Q): 0.44 Prob(JB): 0.00
Heteroskedasticity (H): 1.56 Skew: -0.14
Prob(H) (two-sided): 0.00 Kurtosis: 6.06
===================================================================================
Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
y_pred_arimax110_n225 = arimax_n225.predict(len(y_test), X=X_test_n225.values.reshape(-1, 1))
y_pred_arimax110_n225 = pd.Series(y_pred_arimax110_n225)
y_pred_arimax110_n225.index = y_test.index
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_arimax110_n225.plot(ax=ax, label="ARIMAX(1,1,0) with Nikkei225", linestyle="dashed")
plt.legend()
plt.show()
arima110_n225_metrics = calc_metrics(y_test, y_pred_arimax110_n225, return_dict=True)
Test Error Metrics MAE: 292.0734 MSE: 107173.8052 MAPE: 0.1389 sMAPE: 0.1274 MASE: -12.5497
# Using data before 2020 as training
X_train_oiln225 = tokyo_gas_m.loc[tokyo_gas_m.index.year!=2021, ["oil", "n225"]]
X_test_oiln225 = tokyo_gas_m.loc[tokyo_gas_m.index.year==2021, ["oil", "n225"]]
arimax_oiln225 = auto_arima(y_train, X=X_train_oiln225, start_p=0, d=1, start_q=0, seasonal=False)
print(arimax_oiln225.summary())
SARIMAX Results
==============================================================================
Dep. Variable: y No. Observations: 975
Model: SARIMAX(0, 1, 0) Log Likelihood -4875.807
Date: Thu, 02 Jun 2022 AIC 9757.613
Time: 22:06:03 BIC 9772.258
Sample: 0 HQIC 9763.186
- 975
Covariance Type: opg
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
oil 0.3317 0.626 0.530 0.596 -0.895 1.559
n225 0.0625 0.003 19.055 0.000 0.056 0.069
sigma2 1305.1005 37.129 35.150 0.000 1232.328 1377.873
===================================================================================
Ljung-Box (L1) (Q): 0.62 Jarque-Bera (JB): 385.62
Prob(Q): 0.43 Prob(JB): 0.00
Heteroskedasticity (H): 1.56 Skew: -0.14
Prob(H) (two-sided): 0.00 Kurtosis: 6.07
===================================================================================
Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
y_pred_arimax110_oiln225 = arimax_oiln225.predict(len(y_test), X=X_test_oiln225)
y_pred_arimax110_oiln225 = pd.Series(y_pred_arimax110_oiln225)
y_pred_arimax110_oiln225.index = y_test.index
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_arimax110_oiln225.plot(ax=ax, label="ARIMAX(1,1,0) with Oil Price & Nikkei225", linestyle="dashed")
plt.legend()
plt.show()
arima110_oiln225_metrics = calc_metrics(y_test, y_pred_arimax110_oiln225, return_dict=True)
Test Error Metrics MAE: 298.0109 MSE: 111508.5736 MAPE: 0.1417 sMAPE: 0.1298 MASE: -12.8048
y_pred_arimax110_oiln225 = arimax_oiln225.predict(len(y_test), X=X_test_oiln225)
y_pred_arimax110_oiln225 = pd.Series(y_pred_arimax110_oiln225)
y_pred_arimax110_oiln225.index = y_test.index
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_arima110.plot(ax=ax, label="ARIMA(1,1,0)", linestyle="solid")
y_pred_arimax110_oiln225.plot(ax=ax, label="ARIMAX(1,1,0) with Oil Price & Nikkei225", linestyle="dashed")
y_pred_arimax110_n225.plot(ax=ax, label="ARIMAX(1,1,0) with Nikkei225", linestyle="dashed", color = 'pink')
y_pred_arimax110_oil.plot(ax=ax, label="ARIMAX(1,1,0) with Oil Price", linestyle="dashed")
ax.set_title("ARIMA Model Comparison")
plt.legend()
plt.show()
# Take average of each week
tokyo_gas_weekly = tokyo_gas.resample("W").mean()
# Fill NA with interpolation
tokyo_gas_weekly = tokyo_gas_weekly.interpolate()
# Check stationarity
create_plot_adf_kpss(tokyo_gas_weekly, "tokyo_gas_weekly")
ADF/KPSS tests for tokyo_gas_weekly (alpha=0.05):
stationarity from ADF test: False
stationarity from KPSS test('c'): False
stationarity from KPSS test('ct'): False
| ADF | KPSS('c') | KPSS('ct') | |
|---|---|---|---|
| Test Statistic | -0.666585 | 1.195419 | 0.302181 |
| p-value | 0.855216 | 0.010000 | 0.010000 |
| Lags Used | 4.000000 | 16.000000 | 16.000000 |
| Critical Value (1%) | -3.456569 | 0.739000 | 0.216000 |
| Critical Value (5%) | -2.873079 | 0.463000 | 0.146000 |
| Critical Value (10%) | -2.572919 | 0.347000 | 0.119000 |
# Check seasonality: No seasonality
df_season = pd.DataFrame(tokyo_gas_weekly)
df_season["year"] = df_season.index.year
df_season["week"] = df_season.index.week
# Line plot
fig, ax = plt.subplots(figsize=(15,6))
sns.lineplot("week", "Tokyo Gas", data=df_season, hue="year", ax=ax, legend=None)
ax.grid()
plt.show()
# Check stationarity for differenced data
create_plot_adf_kpss(tokyo_gas_weekly.diff().dropna(), "tokyo_gas_weekly")
ADF/KPSS tests for tokyo_gas_weekly (alpha=0.05):
stationarity from ADF test: True
stationarity from KPSS test('c'): True
stationarity from KPSS test('ct'): True
| ADF | KPSS('c') | KPSS('ct') | |
|---|---|---|---|
| Test Statistic | -9.766553e+00 | 0.161865 | 0.026794 |
| p-value | 7.323609e-17 | 0.100000 | 0.100000 |
| Lags Used | 3.000000e+00 | 16.000000 | 16.000000 |
| Critical Value (1%) | -3.456569e+00 | 0.739000 | 0.216000 |
| Critical Value (5%) | -2.873079e+00 | 0.463000 | 0.146000 |
| Critical Value (10%) | -2.572919e+00 | 0.347000 | 0.119000 |
plot_acf_pacf(tokyo_gas_weekly.diff().dropna(), title = 'correlation check', lags=80)
# GDP
gdp_weekly = gdp_s.resample("W").mean().fillna(method="ffill")
# unemplyment rate
unemp_weekly = unemp_s.resample("W").mean().fillna(method="ffill")
# Oil Price
oil_weekly = oil_s.resample("W").mean()
# JPY/USD
usdjpy_weekly = usdjpy_s.resample("W").mean()
# Interest Rate (long-term)
interest_weekly = interest_s.resample("W").mean()
# Find NAs for interest_weekly
interest_weekly[interest_weekly.isna()]
Date 2019-05-05 NaN Freq: W-SUN, Name: jp_i, dtype: float64
# Interpolation
interest_weekly = interest_weekly.interpolate()
# Nikkei 225 index
n225_weekly = n225_s.resample("W").mean()
# Find NAs
n225_weekly[n225_weekly.isna()]
Date 2019-05-05 NaN Freq: W-SUN, Name: n225, dtype: float64
# Interpolation
n225_weekly = n225_weekly.interpolate()
# Merge data
weekly_m = pd.concat([tokyo_gas_weekly, gdp_weekly, unemp_weekly, oil_weekly, usdjpy_weekly, interest_weekly, n225_weekly], axis=1)
# Show rows with NAs => Need to fill jp_gdp (since last observation for gdp_weekly is for 2021-10-03)
weekly_m[weekly_m.isnull().any(axis=1)]
| Tokyo Gas | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| Date | |||||||
| 2017-01-01 | NaN | 547306.2 | 3.1 | NaN | NaN | NaN | NaN |
| 2021-10-10 | 2050.30 | NaN | 2.8 | 78.504000 | 111.295401 | 0.07200 | 27904.605859 |
| 2021-10-17 | 1939.30 | NaN | 2.8 | 81.176000 | 113.262399 | 0.08740 | 28497.729687 |
| 2021-10-24 | 1907.70 | NaN | 2.8 | 83.476000 | 114.278200 | 0.09280 | 29001.992188 |
| 2021-10-31 | 1910.00 | NaN | 2.8 | 83.844000 | 113.801801 | 0.09940 | 28903.487891 |
| 2021-11-07 | 2021.75 | NaN | 2.9 | 81.788000 | 114.039799 | 0.08300 | 29643.479980 |
| 2021-11-14 | 1990.20 | NaN | 2.9 | 81.930000 | 113.517799 | 0.06840 | 29357.424219 |
| 2021-11-21 | 1949.20 | NaN | 2.9 | 78.992000 | 114.297801 | 0.08020 | 29723.555859 |
| 2021-11-28 | 1997.75 | NaN | 2.9 | 77.793333 | 114.909200 | 0.08475 | 29331.916992 |
| 2021-12-05 | 1959.00 | NaN | 2.8 | 66.890000 | 113.345203 | 0.06980 | 27964.847656 |
| 2021-12-12 | NaN | NaN | NaN | 71.314000 | 113.431400 | 0.05800 | 28481.365625 |
| 2021-12-19 | NaN | NaN | NaN | 71.184000 | 113.724001 | 0.05820 | 28628.970313 |
| 2021-12-26 | NaN | NaN | NaN | 71.625000 | 113.950601 | 0.06920 | 28519.714062 |
| 2022-01-02 | NaN | NaN | NaN | 76.048000 | 114.781801 | 0.07950 | 28861.053223 |
# Drop rows when "Tokyo Gas" is NA
weekly_m = weekly_m.dropna(subset=["Tokyo Gas"])
# Fill NAs for jp_gdp with forward fill
weekly_m["jp_gdp"] = weekly_m["jp_gdp"].fillna(method="ffill")
# Plot external regressors
fig, axes = plt.subplots(2,3, figsize=(16,8), tight_layout=True)
axes_flat = [s for sublist in axes for s in sublist]
for i, c in enumerate(weekly_m.columns.difference(["Tokyo Gas"])):
weekly_m[c].plot(ax=axes_flat[i])
axes_flat[i].set_title(c)
plt.show()
# Using data before June 2021 as training
# One Year Prediction does not generate meaningul outcome, we decided to predict 6 monthes
df_train = weekly_m[:-23]
df_test = weekly_m[-23:]
# extract Tokyo Gas sereis as y_train and y_test
y_train = df_train["Tokyo Gas"]
y_test = df_test["Tokyo Gas"]
# Make prediction with arima model from pmdarima
def predict_from_arima(arima_model, test_y, test_X=None):
y_pred = arima_model.predict(n_periods=len(test_y), X=test_X)
y_pred = pd.Series(y_pred)
y_pred.index = test_y.index
return y_pred
# Make plot fro training/test/prediction
def make_plot(train_y, test_y, pred_y, title):
fig, ax = plt.subplots(figsize=(16,5))
train_y.plot(ax=ax, label="Training")
test_y.plot(ax=ax, label="Test", color="black")
pred_y.plot(ax=ax, label=title, linestyle="dashed")
plt.legend()
plt.show()
arima_uni = auto_arima(y_train, start_p=0, d=1, start_q=0, seasonal=False)
print(arima_uni.summary())
SARIMAX Results
==============================================================================
Dep. Variable: y No. Observations: 234
Model: SARIMAX(2, 1, 2) Log Likelihood -1278.170
Date: Thu, 02 Jun 2022 AIC 2566.341
Time: 22:06:13 BIC 2583.596
Sample: 0 HQIC 2573.299
- 234
Covariance Type: opg
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
ar.L1 0.4466 0.170 2.630 0.009 0.114 0.779
ar.L2 0.3680 0.171 2.158 0.031 0.034 0.702
ma.L1 -0.2512 0.148 -1.694 0.090 -0.542 0.039
ma.L2 -0.6597 0.150 -4.389 0.000 -0.954 -0.365
sigma2 3401.4238 258.999 13.133 0.000 2893.795 3909.053
===================================================================================
Ljung-Box (L1) (Q): 0.01 Jarque-Bera (JB): 50.41
Prob(Q): 0.91 Prob(JB): 0.00
Heteroskedasticity (H): 2.06 Skew: -0.55
Prob(H) (two-sided): 0.00 Kurtosis: 5.00
===================================================================================
Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
print("ARIMA(0,1,1) / No-")
pred_arima011 = predict_from_arima(arima_uni, y_test)
make_plot(y_train, y_test, pred_arima011, "ARIMA(0,1,1) / No-")
metrics_arima011 = calc_metrics(y_test, pred_arima011, return_dict=True)
ARIMA(0,1,1) / No-
Test Error Metrics MAE: 131.9129 MSE: 28228.7658 MAPE: 0.0661 sMAPE: 0.0627 MASE: -1.1125
arimax_n225 = auto_arima(y_train, X=df_train[["n225"]], start_p=0, d=1, start_q=0, seasonal=False)
print(arimax_n225.summary())
SARIMAX Results
==============================================================================
Dep. Variable: y No. Observations: 234
Model: SARIMAX(1, 1, 1) Log Likelihood -1273.925
Date: Thu, 02 Jun 2022 AIC 2555.851
Time: 22:06:20 BIC 2569.655
Sample: 01-08-2017 HQIC 2561.417
- 06-27-2021
Covariance Type: opg
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
n225 0.0303 0.006 4.702 0.000 0.018 0.043
ar.L1 -0.4806 0.168 -2.854 0.004 -0.811 -0.151
ma.L1 0.7074 0.147 4.811 0.000 0.419 0.996
sigma2 3283.1021 205.844 15.950 0.000 2879.656 3686.548
===================================================================================
Ljung-Box (L1) (Q): 0.01 Jarque-Bera (JB): 108.07
Prob(Q): 0.94 Prob(JB): 0.00
Heteroskedasticity (H): 2.35 Skew: -0.63
Prob(H) (two-sided): 0.00 Kurtosis: 6.09
===================================================================================
Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
print("ARIMAX(0,1,1) / n225")
pred_arimax011_n225 = predict_from_arima(arimax_n225, y_test, test_X=df_test[["n225"]])
make_plot(y_train, y_test, pred_arimax011_n225, "ARIMAX(0,1,1)/n225")
metrics_arimax011_n225 = calc_metrics(y_test, pred_arimax011_n225, return_dict=True)
ARIMAX(0,1,1) / n225
Test Error Metrics MAE: 78.5779 MSE: 10447.2298 MAPE: 0.0393 sMAPE: 0.0382 MASE: -0.6627
arimax_oil= auto_arima(y_train, X=df_train[["oil"]], start_p=0, d=1, start_q=0, seasonal=False)
print(arimax_oil.summary())
SARIMAX Results
==============================================================================
Dep. Variable: y No. Observations: 234
Model: SARIMAX(2, 1, 2) Log Likelihood -1278.155
Date: Thu, 02 Jun 2022 AIC 2568.310
Time: 22:06:32 BIC 2589.016
Sample: 01-08-2017 HQIC 2576.659
- 06-27-2021
Covariance Type: opg
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
oil -0.2549 1.125 -0.227 0.821 -2.460 1.950
ar.L1 0.4455 0.169 2.630 0.009 0.114 0.778
ar.L2 0.3680 0.172 2.139 0.032 0.031 0.705
ma.L1 -0.2504 0.148 -1.692 0.091 -0.541 0.040
ma.L2 -0.6600 0.150 -4.390 0.000 -0.955 -0.365
sigma2 3401.0397 259.039 13.129 0.000 2893.332 3908.748
===================================================================================
Ljung-Box (L1) (Q): 0.01 Jarque-Bera (JB): 49.46
Prob(Q): 0.91 Prob(JB): 0.00
Heteroskedasticity (H): 2.05 Skew: -0.54
Prob(H) (two-sided): 0.00 Kurtosis: 4.98
===================================================================================
Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
print("ARIMAX(0,1,1) / oil&usdjpy")
pred_arimax011_oil = predict_from_arima(arimax_oil, y_test, test_X=df_test[["oil"]])
make_plot(y_train, y_test, pred_arimax011_oil, "ARIMAX(0,1,1)/oil")
metrics_arimax011_oil= calc_metrics(y_test, pred_arimax011_oil, return_dict=True)
ARIMAX(0,1,1) / oil&usdjpy
Test Error Metrics MAE: 129.9804 MSE: 27404.4503 MAPE: 0.0652 sMAPE: 0.0618 MASE: -1.0962
arimax_all = auto_arima(y_train, X=df_train[df_train.columns.difference(["Tokyo Gas"])], start_p=0, d=1, start_q=0, seasonal=False)
print(arimax_all.summary())
SARIMAX Results
==============================================================================
Dep. Variable: y No. Observations: 234
Model: SARIMAX(1, 1, 1) Log Likelihood -1271.564
Date: Thu, 02 Jun 2022 AIC 2561.128
Time: 22:06:42 BIC 2592.187
Sample: 01-08-2017 HQIC 2573.652
- 06-27-2021
Covariance Type: opg
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
jp_gdp -0.0020 0.001 -2.732 0.006 -0.004 -0.001
jp_i 20.2221 183.043 0.110 0.912 -338.535 378.980
jp_unemp -3.8768 68.513 -0.057 0.955 -138.160 130.406
n225 0.0342 0.010 3.308 0.001 0.014 0.054
oil -2.7543 1.154 -2.386 0.017 -5.017 -0.492
usdjpy 3.3189 5.539 0.599 0.549 -7.537 14.175
ar.L1 -0.4625 0.207 -2.229 0.026 -0.869 -0.056
ma.L1 0.6683 0.190 3.514 0.000 0.295 1.041
sigma2 3254.4271 224.249 14.513 0.000 2814.907 3693.947
===================================================================================
Ljung-Box (L1) (Q): 0.03 Jarque-Bera (JB): 100.90
Prob(Q): 0.86 Prob(JB): 0.00
Heteroskedasticity (H): 2.23 Skew: -0.67
Prob(H) (two-sided): 0.00 Kurtosis: 5.93
===================================================================================
Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
print("ARIMAX(1,1,1) / all ")
pred_arimax011_all = predict_from_arima(arimax_all, y_test, test_X=df_test[df_test.columns.difference(["Tokyo Gas"])])
make_plot(y_train, y_test, pred_arimax011_all, "ARIMAX(1,1,1)/all ")
metrics_arimax011_all = calc_metrics(y_test, pred_arimax011_all, return_dict=True)
ARIMAX(1,1,1) / all
Test Error Metrics MAE: 77.2154 MSE: 10671.9102 MAPE: 0.0388 sMAPE: 0.0375 MASE: -0.6512
fig, ax = plt.subplots(figsize=(16,6))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
pred_arimax011_all.plot(ax=ax, label="ARIMAX(1,1,1)/all exor", linestyle="dashed", color = 'red')
pred_arimax011_oil.plot(ax=ax, label="ARIMAX(1,1,1)/oil", linestyle="dashed")
pred_arimax011_n225.plot(ax=ax, label="ARIMAX(1,1,1)/n225", linestyle="dashed" )
plt.legend()
plt.show()
#Accuracy Check
weekly_pred = dict()
weekly_metrics = dict()
weekly_pred["ARIMA(0,1,1) / No-"] = pred_arima011
weekly_metrics["ARIMA(0,1,1) / No-"] = metrics_arima011
weekly_pred["ARIMAX(0,1,1) / all"] = pred_arimax011_all
weekly_metrics["ARIMAX(0,1,1) / all"] = metrics_arimax011_all
weekly_pred["ARIMAX(0,1,1) / n225"] = pred_arimax011_n225
weekly_metrics["ARIMAX(0,1,1) / n225"] = metrics_arimax011_n225
weekly_pred["ARIMAX(0,1,1) / oil"] = pred_arimax011_oil
weekly_metrics["ARIMAX(0,1,1) / oil "] = metrics_arimax011_oil
pd.DataFrame(weekly_metrics).T
| MAE | MSE | MAPE | sMAPE | MASE | |
|---|---|---|---|---|---|
| ARIMA(0,1,1) / No- | 131.9129 | 28228.7658 | 0.0661 | 0.0627 | -1.1125 |
| ARIMAX(0,1,1) / all | 77.2154 | 10671.9102 | 0.0388 | 0.0375 | -0.6512 |
| ARIMAX(0,1,1) / n225 | 78.5779 | 10447.2298 | 0.0393 | 0.0382 | -0.6627 |
| ARIMAX(0,1,1) / oil | 129.9804 | 27404.4503 | 0.0652 | 0.0618 | -1.0962 |
tokyo = main5.loc[main5["Name"]=="Tokyo Gas", "Adjusted Close"]
tokyo = tokyo.dropna()
tokyo.name = "tokyo"
tokyo = pd.DataFrame(tokyo)
tokyo.reset_index()
| Date | tokyo | |
|---|---|---|
| 0 | 2017-01-04 | 2691.5 |
| 1 | 2017-01-05 | 2683.5 |
| 2 | 2017-01-06 | 2691.5 |
| 3 | 2017-01-10 | 2628.0 |
| 4 | 2017-01-11 | 2648.5 |
| ... | ... | ... |
| 1196 | 2021-11-29 | 1993.0 |
| 1197 | 2021-11-30 | 1945.0 |
| 1198 | 2021-12-01 | 1904.0 |
| 1199 | 2021-12-02 | 1975.0 |
| 1200 | 2021-12-03 | 1978.0 |
1201 rows × 2 columns
# Check stationarity
create_plot_adf_kpss(tokyo, "Tokyo_original")
ADF/KPSS tests for Tokyo_original (alpha=0.05):
stationarity from ADF test: False
stationarity from KPSS test('c'): False
stationarity from KPSS test('ct'): False
| ADF | KPSS('c') | KPSS('ct') | |
|---|---|---|---|
| Test Statistic | -1.201363 | 3.438455 | 0.722794 |
| p-value | 0.673009 | 0.010000 | 0.010000 |
| Lags Used | 9.000000 | 23.000000 | 23.000000 |
| Critical Value (1%) | -3.435852 | 0.739000 | 0.216000 |
| Critical Value (5%) | -2.863970 | 0.463000 | 0.146000 |
| Critical Value (10%) | -2.568064 | 0.347000 | 0.119000 |
# Take first difference
tokyo_diff = tokyo.diff().dropna()
# Check stationarity
create_plot_adf_kpss(tokyo_diff, "Tokyo (diff(1))")
ADF/KPSS tests for Tokyo (diff(1)) (alpha=0.05):
stationarity from ADF test: True
stationarity from KPSS test('c'): True
stationarity from KPSS test('ct'): True
| ADF | KPSS('c') | KPSS('ct') | |
|---|---|---|---|
| Test Statistic | -1.315887e+01 | 0.089789 | 0.0155 |
| p-value | 1.319894e-24 | 0.100000 | 0.1000 |
| Lags Used | 8.000000e+00 | 23.000000 | 23.0000 |
| Critical Value (1%) | -3.435852e+00 | 0.739000 | 0.2160 |
| Critical Value (5%) | -2.863970e+00 | 0.463000 | 0.1460 |
| Critical Value (10%) | -2.568064e+00 | 0.347000 | 0.1190 |
# Using data before 2020 as training
y_train = tokyo[tokyo.index.year!=2021]
y_test = tokyo[tokyo.index.year==2021]
y_train
| tokyo | |
|---|---|
| Date | |
| 2017-01-04 | 2691.5 |
| 2017-01-05 | 2683.5 |
| 2017-01-06 | 2691.5 |
| 2017-01-10 | 2628.0 |
| 2017-01-11 | 2648.5 |
| ... | ... |
| 2020-12-24 | 2308.5 |
| 2020-12-25 | 2294.5 |
| 2020-12-28 | 2344.5 |
| 2020-12-29 | 2374.5 |
| 2020-12-30 | 2384.0 |
975 rows × 1 columns
# Merge data and fill NAs with interpolation
tokyo_1 = pd.concat([tokyo, oil_s], axis=1)
tokyo_1 = tokyo_1.dropna(subset=["tokyo"])
tokyo_1["oil"] = tokyo_1["oil"].interpolate()
tokyo_1
| tokyo | oil | |
|---|---|---|
| Date | ||
| 2017-01-04 | 2691.5 | 53.26 |
| 2017-01-05 | 2683.5 | 53.77 |
| 2017-01-06 | 2691.5 | 53.98 |
| 2017-01-10 | 2628.0 | 50.82 |
| 2017-01-11 | 2648.5 | 52.19 |
| ... | ... | ... |
| 2021-11-29 | 1993.0 | 69.88 |
| 2021-11-30 | 1945.0 | 66.14 |
| 2021-12-01 | 1904.0 | 65.44 |
| 2021-12-02 | 1975.0 | 66.60 |
| 2021-12-03 | 1978.0 | 66.39 |
1201 rows × 2 columns
# Take first difference
tokyo_1_diff = tokyo_1.diff().dropna()
# Split into train/test
train_diff1 = tokyo_1_diff[tokyo_1_diff.index.year!=2021]
test_diff1 = tokyo_1_diff[tokyo_1_diff.index.year==2021]
from statsmodels.tsa.vector_ar.var_model import VAR
var_model1 = VAR(train_diff1)
results_aic1 =[]
for p in range(1,21):
results1 = var_model1.fit(p)
results_aic1.append(results1.aic)
# Min=2
plt.plot(list(np.arange(1,21,1)), results_aic1, marker="o")
plt.xlabel("Order")
plt.ylabel("AIC")
plt.show()
var2_model1 = var_model1.fit(6)
print(var2_model1.summary())
Summary of Regression Results
==================================
Model: VAR
Method: OLS
Date: Thu, 02, Jun, 2022
Time: 20:54:00
--------------------------------------------------------------------
No. of Equations: 2.00000 BIC: 9.25338
Nobs: 968.000 HQIC: 9.17228
Log likelihood: -7136.32 FPE: 9158.52
AIC: 9.12244 Det(Omega_mle): 8917.39
--------------------------------------------------------------------
Results for equation tokyo
===========================================================================
coefficient std. error t-stat prob
---------------------------------------------------------------------------
const -0.340061 1.246938 -0.273 0.785
L1.tokyo 0.022771 0.032311 0.705 0.481
L1.oil -0.386131 0.513725 -0.752 0.452
L2.tokyo -0.049426 0.032070 -1.541 0.123
L2.oil -0.638941 0.553575 -1.154 0.248
L3.tokyo -0.083926 0.032066 -2.617 0.009
L3.oil -0.648369 0.562825 -1.152 0.249
L4.tokyo -0.080839 0.032012 -2.525 0.012
L4.oil 0.452125 0.562477 0.804 0.422
L5.tokyo 0.102002 0.032073 3.180 0.001
L5.oil 0.862463 0.553982 1.557 0.120
L6.tokyo -0.036055 0.032241 -1.118 0.263
L6.oil -0.443726 0.514118 -0.863 0.388
===========================================================================
Results for equation oil
===========================================================================
coefficient std. error t-stat prob
---------------------------------------------------------------------------
const -0.009125 0.078336 -0.116 0.907
L1.tokyo -0.001432 0.002030 -0.705 0.481
L1.oil -0.401247 0.032273 -12.433 0.000
L2.tokyo 0.002468 0.002015 1.225 0.221
L2.oil -0.199741 0.034777 -5.743 0.000
L3.tokyo -0.002999 0.002014 -1.489 0.137
L3.oil -0.095167 0.035358 -2.692 0.007
L4.tokyo -0.000974 0.002011 -0.484 0.628
L4.oil -0.040897 0.035336 -1.157 0.247
L5.tokyo 0.000810 0.002015 0.402 0.688
L5.oil 0.026968 0.034802 0.775 0.438
L6.tokyo -0.002688 0.002025 -1.327 0.185
L6.oil 0.069058 0.032298 2.138 0.033
===========================================================================
Correlation matrix of residuals
tokyo oil
tokyo 1.000000 0.035334
oil 0.035334 1.000000
# Calculate yhat
yhat_var2_1 = var2_model1.forecast(var_model1.endog, steps=len(test_diff1))
# clean up yhat
df_yhat_var2_1 = pd.DataFrame(yhat_var2_1)
df_yhat_var2_1.index = test_diff1.index
df_yhat_var2_1.columns = test_diff1.columns
df_yhat_var2_1
| tokyo | oil | |
|---|---|---|
| Date | ||
| 2021-01-04 | -4.650032 | -0.247341 |
| 2021-01-05 | -10.324048 | -0.155952 |
| 2021-01-06 | 2.018371 | 0.042515 |
| 2021-01-07 | 1.968440 | -0.135240 |
| 2021-01-08 | 0.930132 | 0.061009 |
| ... | ... | ... |
| 2021-11-29 | -0.298810 | -0.004684 |
| 2021-11-30 | -0.298810 | -0.004684 |
| 2021-12-01 | -0.298810 | -0.004684 |
| 2021-12-02 | -0.298810 | -0.004684 |
| 2021-12-03 | -0.298810 | -0.004684 |
226 rows × 2 columns
# Recover first order differencing
df_pred_var2_1 = pd.concat([tokyo_1[tokyo_1.index==pd.datetime(2020,12,30)], df_yhat_var2_1], axis=0).cumsum()
df_pred_var2_1 = df_pred_var2_1[df_pred_var2_1.index!=pd.datetime(2020,12,30)]
y_pred_var2_1 = df_pred_var2_1["tokyo"]
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_var2_1.plot(ax=ax, label="VAR(6) with Oil Price", linestyle="dashed")
plt.legend()
plt.show()
var2_metrics1 = calc_metrics(y_test, y_pred_var2_1, return_dict=True)
Test Error Metrics MAE: 173.6549 MSE: 42873.2797 MAPE: 0.0833 sMAPE: 0.0786 MASE: tokyo -7.4615 dtype: float64
# Merge data and fill NAs with interpolation
tokyo_2 = pd.concat([tokyo, oil_s, n225_s], axis=1)
tokyo_2 = tokyo_2.dropna(subset=["tokyo"])
tokyo_2["oil"] = tokyo_2["oil"].interpolate()
tokyo_2["n225"] = tokyo_2["n225"].interpolate()
tokyo_2
| tokyo | oil | n225 | |
|---|---|---|---|
| Date | |||
| 2017-01-04 | 2691.5 | 53.26 | 19594.160156 |
| 2017-01-05 | 2683.5 | 53.77 | 19520.689453 |
| 2017-01-06 | 2691.5 | 53.98 | 19454.330078 |
| 2017-01-10 | 2628.0 | 50.82 | 19301.439453 |
| 2017-01-11 | 2648.5 | 52.19 | 19364.669922 |
| ... | ... | ... | ... |
| 2021-11-29 | 1993.0 | 69.88 | 28283.919922 |
| 2021-11-30 | 1945.0 | 66.14 | 27821.759766 |
| 2021-12-01 | 1904.0 | 65.44 | 27935.619141 |
| 2021-12-02 | 1975.0 | 66.60 | 27753.369141 |
| 2021-12-03 | 1978.0 | 66.39 | 28029.570312 |
1201 rows × 3 columns
# Take first difference
tokyo_2_diff = tokyo_2.diff().dropna()
# Split into train/test
train_diff2 = tokyo_2_diff[tokyo_2_diff.index.year!=2021]
test_diff2 = tokyo_2_diff[tokyo_2_diff.index.year==2021]
from statsmodels.tsa.vector_ar.var_model import VAR
var_model2 = VAR(train_diff2)
results_aic2 =[]
for p in range(1,21):
results2 = var_model2.fit(p)
results_aic2.append(results2.aic)
# Min=2
plt.plot(list(np.arange(1,21,1)), results_aic2, marker="o")
plt.xlabel("Order")
plt.ylabel("AIC")
plt.show()
var2_model2 = var_model2.fit(20)
print(var2_model2.summary())
Summary of Regression Results
==================================
Model: VAR
Method: OLS
Date: Wed, 01, Jun, 2022
Time: 18:34:47
--------------------------------------------------------------------
No. of Equations: 3.00000 BIC: 20.8181
Nobs: 954.000 HQIC: 20.2408
Log likelihood: -13363.5 FPE: 4.32974e+08
AIC: 19.8857 Det(Omega_mle): 3.59509e+08
--------------------------------------------------------------------
Results for equation tokyo
============================================================================
coefficient std. error t-stat prob
----------------------------------------------------------------------------
const 0.351132 1.258189 0.279 0.780
L1.tokyo 0.015093 0.036327 0.415 0.678
L1.oil -0.542846 0.558643 -0.972 0.331
L1.n225 -0.002542 0.005798 -0.438 0.661
L2.tokyo -0.064720 0.036285 -1.784 0.074
L2.oil -0.454756 0.602104 -0.755 0.450
L2.n225 0.002616 0.005812 0.450 0.653
L3.tokyo -0.100397 0.036229 -2.771 0.006
L3.oil -0.015104 0.612118 -0.025 0.980
L3.n225 0.006219 0.005833 1.066 0.286
L4.tokyo -0.095096 0.036387 -2.613 0.009
L4.oil 0.948821 0.615096 1.543 0.123
L4.n225 0.002262 0.005842 0.387 0.699
L5.tokyo 0.098587 0.036411 2.708 0.007
L5.oil 1.129825 0.616268 1.833 0.067
L5.n225 -0.003429 0.005845 -0.587 0.557
L6.tokyo -0.081797 0.036388 -2.248 0.025
L6.oil -0.062030 0.615274 -0.101 0.920
L6.n225 0.006804 0.005881 1.157 0.247
L7.tokyo 0.032459 0.036406 0.892 0.373
L7.oil 0.680964 0.614076 1.109 0.267
L7.n225 -0.008665 0.005890 -1.471 0.141
L8.tokyo -0.037013 0.036476 -1.015 0.310
L8.oil -0.436675 0.615356 -0.710 0.478
L8.n225 -0.013087 0.005909 -2.215 0.027
L9.tokyo -0.031780 0.036575 -0.869 0.385
L9.oil 0.414432 0.611860 0.677 0.498
L9.n225 -0.009782 0.005930 -1.650 0.099
L10.tokyo -0.002452 0.036424 -0.067 0.946
L10.oil 0.401559 0.605260 0.663 0.507
L10.n225 -0.003027 0.005930 -0.510 0.610
L11.tokyo 0.056379 0.036410 1.548 0.122
L11.oil -0.146303 0.605980 -0.241 0.809
L11.n225 -0.009915 0.005935 -1.671 0.095
L12.tokyo -0.034663 0.036572 -0.948 0.343
L12.oil -0.611684 0.608674 -1.005 0.315
L12.n225 0.004667 0.005926 0.788 0.431
L13.tokyo -0.006824 0.036293 -0.188 0.851
L13.oil -0.758233 0.610644 -1.242 0.214
L13.n225 -0.002400 0.005905 -0.406 0.684
L14.tokyo 0.067055 0.036180 1.853 0.064
L14.oil -0.238576 0.610996 -0.390 0.696
L14.n225 -0.019122 0.005881 -3.251 0.001
L15.tokyo 0.092803 0.036178 2.565 0.010
L15.oil 0.437240 0.609467 0.717 0.473
L15.n225 -0.020689 0.005889 -3.513 0.000
L16.tokyo -0.083095 0.036116 -2.301 0.021
L16.oil 1.153692 0.602314 1.915 0.055
L16.n225 0.008427 0.005931 1.421 0.155
L17.tokyo -0.027478 0.036371 -0.755 0.450
L17.oil 0.386530 0.601252 0.643 0.520
L17.n225 -0.004870 0.005923 -0.822 0.411
L18.tokyo -0.027572 0.036377 -0.758 0.448
L18.oil 0.089415 0.595950 0.150 0.881
L18.n225 0.009880 0.005925 1.667 0.095
L19.tokyo -0.039782 0.036341 -1.095 0.274
L19.oil -0.179430 0.581900 -0.308 0.758
L19.n225 0.009633 0.005934 1.623 0.105
L20.tokyo -0.021900 0.036689 -0.597 0.551
L20.oil -0.277037 0.533856 -0.519 0.604
L20.n225 -0.013835 0.005961 -2.321 0.020
============================================================================
Results for equation oil
============================================================================
coefficient std. error t-stat prob
----------------------------------------------------------------------------
const -0.048632 0.075640 -0.643 0.520
L1.tokyo 0.000459 0.002184 0.210 0.834
L1.oil -0.386837 0.033584 -11.518 0.000
L1.n225 -0.000450 0.000349 -1.292 0.196
L2.tokyo 0.000029 0.002181 0.013 0.989
L2.oil -0.217058 0.036197 -5.997 0.000
L2.n225 0.000973 0.000349 2.785 0.005
L3.tokyo -0.003587 0.002178 -1.647 0.100
L3.oil -0.121988 0.036799 -3.315 0.001
L3.n225 0.000523 0.000351 1.492 0.136
L4.tokyo -0.002255 0.002187 -1.031 0.303
L4.oil -0.067169 0.036978 -1.816 0.069
L4.n225 0.000081 0.000351 0.231 0.818
L5.tokyo -0.002069 0.002189 -0.945 0.345
L5.oil -0.006405 0.037049 -0.173 0.863
L5.n225 0.001268 0.000351 3.608 0.000
L6.tokyo -0.003590 0.002188 -1.641 0.101
L6.oil 0.055456 0.036989 1.499 0.134
L6.n225 0.000682 0.000354 1.929 0.054
L7.tokyo 0.000231 0.002189 0.105 0.916
L7.oil -0.000465 0.036917 -0.013 0.990
L7.n225 0.000733 0.000354 2.071 0.038
L8.tokyo -0.003090 0.002193 -1.409 0.159
L8.oil -0.014258 0.036994 -0.385 0.700
L8.n225 0.000437 0.000355 1.229 0.219
L9.tokyo 0.001118 0.002199 0.509 0.611
L9.oil -0.004411 0.036784 -0.120 0.905
L9.n225 0.000171 0.000357 0.478 0.632
L10.tokyo 0.000171 0.002190 0.078 0.938
L10.oil -0.055680 0.036387 -1.530 0.126
L10.n225 -0.000211 0.000356 -0.592 0.554
L11.tokyo -0.005921 0.002189 -2.705 0.007
L11.oil -0.088633 0.036430 -2.433 0.015
L11.n225 0.000960 0.000357 2.689 0.007
L12.tokyo -0.000430 0.002199 -0.196 0.845
L12.oil -0.085505 0.036592 -2.337 0.019
L12.n225 0.000291 0.000356 0.817 0.414
L13.tokyo 0.000204 0.002182 0.094 0.925
L13.oil 0.019311 0.036711 0.526 0.599
L13.n225 0.000809 0.000355 2.278 0.023
L14.tokyo 0.000654 0.002175 0.300 0.764
L14.oil -0.014291 0.036732 -0.389 0.697
L14.n225 -0.000225 0.000354 -0.636 0.525
L15.tokyo -0.002266 0.002175 -1.042 0.298
L15.oil 0.026914 0.036640 0.735 0.463
L15.n225 -0.000203 0.000354 -0.574 0.566
L16.tokyo -0.008418 0.002171 -3.877 0.000
L16.oil 0.066019 0.036210 1.823 0.068
L16.n225 -0.000112 0.000357 -0.314 0.754
L17.tokyo 0.005917 0.002187 2.706 0.007
L17.oil 0.010854 0.036146 0.300 0.764
L17.n225 0.000643 0.000356 1.804 0.071
L18.tokyo -0.003321 0.002187 -1.518 0.129
L18.oil -0.012463 0.035827 -0.348 0.728
L18.n225 -0.001677 0.000356 -4.709 0.000
L19.tokyo 0.008701 0.002185 3.983 0.000
L19.oil -0.008770 0.034983 -0.251 0.802
L19.n225 -0.001238 0.000357 -3.470 0.001
L20.tokyo 0.000461 0.002206 0.209 0.834
L20.oil 0.039802 0.032094 1.240 0.215
L20.n225 0.000699 0.000358 1.950 0.051
============================================================================
Results for equation n225
============================================================================
coefficient std. error t-stat prob
----------------------------------------------------------------------------
const 10.241918 7.960552 1.287 0.198
L1.tokyo 0.017281 0.229838 0.075 0.940
L1.oil 12.224234 3.534529 3.459 0.001
L1.n225 -0.011759 0.036685 -0.321 0.749
L2.tokyo -0.317532 0.229574 -1.383 0.167
L2.oil 13.104383 3.809508 3.440 0.001
L2.n225 0.075546 0.036772 2.054 0.040
L3.tokyo -0.529180 0.229219 -2.309 0.021
L3.oil 7.733229 3.872866 1.997 0.046
L3.n225 0.036308 0.036905 0.984 0.325
L4.tokyo -0.134185 0.230217 -0.583 0.560
L4.oil 9.623173 3.891711 2.473 0.013
L4.n225 0.050597 0.036965 1.369 0.171
L5.tokyo 0.204385 0.230373 0.887 0.375
L5.oil 1.819025 3.899121 0.467 0.641
L5.n225 -0.025090 0.036983 -0.678 0.497
L6.tokyo -0.273477 0.230225 -1.188 0.235
L6.oil 0.825211 3.892837 0.212 0.832
L6.n225 -0.030005 0.037211 -0.806 0.420
L7.tokyo 0.625427 0.230339 2.715 0.007
L7.oil -7.693255 3.885257 -1.980 0.048
L7.n225 -0.087963 0.037267 -2.360 0.018
L8.tokyo 0.136385 0.230784 0.591 0.555
L8.oil 5.373894 3.893356 1.380 0.168
L8.n225 -0.065004 0.037389 -1.739 0.082
L9.tokyo 0.150621 0.231409 0.651 0.515
L9.oil 7.570115 3.871231 1.955 0.051
L9.n225 0.019005 0.037519 0.507 0.612
L10.tokyo -0.152613 0.230453 -0.662 0.508
L10.oil -2.203638 3.829478 -0.575 0.565
L10.n225 0.055288 0.037517 1.474 0.141
L11.tokyo 0.384781 0.230368 1.670 0.095
L11.oil -6.074745 3.834033 -1.584 0.113
L11.n225 -0.041679 0.037551 -1.110 0.267
L12.tokyo 0.050651 0.231393 0.219 0.827
L12.oil -0.564994 3.851079 -0.147 0.883
L12.n225 0.014717 0.037495 0.393 0.695
L13.tokyo 0.051655 0.229626 0.225 0.822
L13.oil 2.866915 3.863538 0.742 0.458
L13.n225 0.032218 0.037362 0.862 0.389
L14.tokyo 0.571922 0.228911 2.498 0.012
L14.oil 6.297827 3.865770 1.629 0.103
L14.n225 -0.062885 0.037209 -1.690 0.091
L15.tokyo 0.236120 0.228899 1.032 0.302
L15.oil 6.555963 3.856092 1.700 0.089
L15.n225 -0.034248 0.037259 -0.919 0.358
L16.tokyo -0.272733 0.228503 -1.194 0.233
L16.oil 6.579832 3.810839 1.727 0.084
L16.n225 -0.014039 0.037527 -0.374 0.708
L17.tokyo 0.040175 0.230119 0.175 0.861
L17.oil 1.508583 3.804118 0.397 0.692
L17.n225 0.000349 0.037474 0.009 0.993
L18.tokyo -0.453015 0.230155 -1.968 0.049
L18.oil -3.018696 3.770569 -0.801 0.423
L18.n225 0.010712 0.037487 0.286 0.775
L19.tokyo -0.334998 0.229927 -1.457 0.145
L19.oil -3.806185 3.681677 -1.034 0.301
L19.n225 0.034964 0.037545 0.931 0.352
L20.tokyo 0.217767 0.232130 0.938 0.348
L20.oil 1.949594 3.377705 0.577 0.564
L20.n225 -0.062460 0.037718 -1.656 0.098
============================================================================
Correlation matrix of residuals
tokyo oil n225
tokyo 1.000000 0.071436 0.401044
oil 0.071436 1.000000 0.121275
n225 0.401044 0.121275 1.000000
# Calculate yhat
yhat_var2_2 = var2_model2.forecast(var_model2.endog, steps=len(test_diff2))
# clean up yhat
df_yhat_var2_2 = pd.DataFrame(yhat_var2_2)
df_yhat_var2_2.index = test_diff2.index
df_yhat_var2_2.columns = test_diff2.columns
df_yhat_var2_2
| tokyo | oil | n225 | |
|---|---|---|---|
| Date | |||
| 2021-01-04 | 11.688611 | 0.853693 | 99.427897 |
| 2021-01-05 | 2.689187 | -0.254569 | 28.897992 |
| 2021-01-06 | 1.638704 | -1.446170 | 19.718546 |
| 2021-01-07 | -12.107077 | 1.342011 | -37.381688 |
| 2021-01-08 | -0.264490 | 1.536821 | -17.597606 |
| ... | ... | ... | ... |
| 2021-11-29 | -0.158020 | -0.004643 | 8.979042 |
| 2021-11-30 | -0.158013 | -0.004643 | 8.979046 |
| 2021-12-01 | -0.158008 | -0.004643 | 8.979024 |
| 2021-12-02 | -0.158005 | -0.004644 | 8.978982 |
| 2021-12-03 | -0.158006 | -0.004644 | 8.978933 |
226 rows × 3 columns
# Recover first order differencing
df_pred_var2_2 = pd.concat([tokyo_2[tokyo_2.index==pd.datetime(2020,12,30)], df_yhat_var2_2], axis=0).cumsum()
df_pred_var2_2 = df_pred_var2_2[df_pred_var2_2.index!=pd.datetime(2020,12,30)]
y_pred_var2_2 = df_pred_var2_2["tokyo"]
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_var2_2.plot(ax=ax, label="VAR(20) with Oil Price & Nikkei225", linestyle="dashed")
plt.legend()
plt.show()
var2_metrics2 = calc_metrics(y_test, y_pred_var2_2, return_dict=True)
Test Error Metrics MAE: 164.9285 MSE: 40503.7034 MAPE: 0.0793 sMAPE: 0.0749 MASE: -7.0866
# Merge data and fill NAs with interpolation
tokyo_3 = pd.concat([tokyo, oil_s, n225_s, unemp_s], axis=1)
tokyo_3["jp_unemp"] = tokyo_3["jp_unemp"].fillna(method="ffill")
tokyo_3 = tokyo_3.dropna(subset=["tokyo"])
tokyo_3["oil"] = tokyo_3["oil"].interpolate()
tokyo_3["n225"] = tokyo_3["n225"].interpolate()
tokyo_3[tokyo_3.isnull().any(axis=1)]
| tokyo | oil | n225 | jp_unemp | |
|---|---|---|---|---|
| Date |
# Take first difference
tokyo_3_diff = tokyo_3.diff().dropna()
# Split into train/test
train_diff3 = tokyo_3_diff[tokyo_3_diff.index.year!=2021]
test_diff3 = tokyo_3_diff[tokyo_3_diff.index.year==2021]
from statsmodels.tsa.vector_ar.var_model import VAR
var_model3 = VAR(train_diff3)
results_aic3 =[]
for p in range(1,21):
results3 = var_model3.fit(p)
results_aic3.append(results3.aic)
# Min=2
plt.plot(list(np.arange(1,21,1)), results_aic3, marker="o")
plt.xlabel("Order")
plt.ylabel("AIC")
plt.show()
var2_model3 = var_model3.fit(2)
print(var2_model3.summary())
Summary of Regression Results
==================================
Model: VAR
Method: OLS
Date: Wed, 01, Jun, 2022
Time: 11:41:01
--------------------------------------------------------------------
No. of Equations: 4.00000 BIC: 13.1836
Nobs: 972.000 HQIC: 13.0717
Log likelihood: -11800.2 FPE: 443700.
AIC: 13.0029 Det(Omega_mle): 427641.
--------------------------------------------------------------------
Results for equation tokyo
==============================================================================
coefficient std. error t-stat prob
------------------------------------------------------------------------------
const -0.375938 1.265658 -0.297 0.766
L1.tokyo 0.016420 0.035015 0.469 0.639
L1.oil -0.335657 0.514277 -0.653 0.514
L1.n225 0.000997 0.005647 0.176 0.860
L1.jp_unemp -10.421555 41.499732 -0.251 0.802
L2.tokyo -0.061461 0.035011 -1.756 0.079
L2.oil -0.502852 0.513760 -0.979 0.328
L2.n225 0.003756 0.005655 0.664 0.507
L2.jp_unemp 25.282122 41.490865 0.609 0.542
==============================================================================
Results for equation oil
==============================================================================
coefficient std. error t-stat prob
------------------------------------------------------------------------------
const -0.009462 0.078454 -0.121 0.904
L1.tokyo 0.000509 0.002170 0.235 0.815
L1.oil -0.377779 0.031878 -11.851 0.000
L1.n225 -0.000589 0.000350 -1.683 0.092
L1.jp_unemp 0.138423 2.572431 0.054 0.957
L2.tokyo 0.000794 0.002170 0.366 0.715
L2.oil -0.160489 0.031846 -5.039 0.000
L2.n225 0.000744 0.000351 2.122 0.034
L2.jp_unemp 0.569405 2.571881 0.221 0.825
==============================================================================
Results for equation n225
==============================================================================
coefficient std. error t-stat prob
------------------------------------------------------------------------------
const 7.362547 7.836165 0.940 0.347
L1.tokyo -0.122056 0.216791 -0.563 0.573
L1.oil 9.626528 3.184081 3.023 0.003
L1.n225 0.017272 0.034963 0.494 0.621
L1.jp_unemp 70.749315 256.940443 0.275 0.783
L2.tokyo -0.396456 0.216764 -1.829 0.067
L2.oil 10.037418 3.180882 3.156 0.002
L2.n225 0.086823 0.035013 2.480 0.013
L2.jp_unemp -34.260254 256.885540 -0.133 0.894
==============================================================================
Results for equation jp_unemp
==============================================================================
coefficient std. error t-stat prob
------------------------------------------------------------------------------
const 0.000131 0.000987 0.132 0.895
L1.tokyo -0.000003 0.000027 -0.095 0.925
L1.oil 0.000297 0.000401 0.742 0.458
L1.n225 0.000001 0.000004 0.145 0.885
L1.jp_unemp 0.000608 0.032353 0.019 0.985
L2.tokyo 0.000003 0.000027 0.121 0.904
L2.oil 0.000070 0.000401 0.176 0.860
L2.n225 -0.000004 0.000004 -0.916 0.360
L2.jp_unemp -0.002637 0.032346 -0.082 0.935
==============================================================================
Correlation matrix of residuals
tokyo oil n225 jp_unemp
tokyo 1.000000 0.045149 0.399272 -0.068800
oil 0.045149 1.000000 0.110902 -0.004741
n225 0.399272 0.110902 1.000000 -0.085350
jp_unemp -0.068800 -0.004741 -0.085350 1.000000
# Calculate yhat
yhat_var2_3 = var2_model3.forecast(var_model3.endog, steps=len(test_diff3))
# clean up yhat
df_yhat_var2_3 = pd.DataFrame(yhat_var2_3)
df_yhat_var2_3.index = test_diff3.index
df_yhat_var2_3.columns = test_diff3.columns
df_yhat_var2_3
| tokyo | oil | n225 | jp_unemp | |
|---|---|---|---|---|
| Date | ||||
| 2021-01-04 | 0.188015 | 0.419958 | 61.437427 | -0.002616 |
| 2021-01-05 | -1.670987 | -0.351860 | 1.642294 | 0.000852 |
| 2021-01-06 | -0.350625 | 0.098725 | 13.832494 | -0.000179 |
| 2021-01-07 | -0.091842 | 0.001737 | 5.826075 | 0.000130 |
| 2021-01-08 | -0.354250 | -0.019515 | 9.837397 | 0.000086 |
| ... | ... | ... | ... | ... |
| 2021-11-29 | -0.316197 | -0.005540 | 8.283459 | 0.000100 |
| 2021-11-30 | -0.316197 | -0.005540 | 8.283459 | 0.000100 |
| 2021-12-01 | -0.316197 | -0.005540 | 8.283459 | 0.000100 |
| 2021-12-02 | -0.316197 | -0.005540 | 8.283459 | 0.000100 |
| 2021-12-03 | -0.316197 | -0.005540 | 8.283459 | 0.000100 |
226 rows × 4 columns
# Recover first order differencing
df_pred_var2_3 = pd.concat([tokyo_3[tokyo_3.index==pd.datetime(2020,12,30)], df_yhat_var2_3], axis=0).cumsum()
df_pred_var2_3 = df_pred_var2_3[df_pred_var2_3.index!=pd.datetime(2020,12,30)]
y_pred_var2_3 = df_pred_var2_3["tokyo"]
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_var2_3.plot(ax=ax, label="VAR(2) with Oil, N225, Unemployement", linestyle="dashed")
plt.legend()
plt.show()
var2_metrics3 = calc_metrics(y_test, y_pred_var2_3, return_dict=True)
Test Error Metrics MAE: 179.2476 MSE: 44981.3307 MAPE: 0.0859 sMAPE: 0.081 MASE: -7.7018
# Merge data and fill NAs with interpolation
tokyo_4 = pd.concat([tokyo, oil_s, n225_s, unemp_s, gdp_s], axis=1)
tokyo_4["jp_gdp"] = tokyo_4["jp_gdp"].fillna(method="ffill")
tokyo_4["jp_unemp"] = tokyo_4["jp_unemp"].fillna(method="ffill")
tokyo_4 = tokyo_4.dropna(subset=["tokyo"])
tokyo_4["oil"] = tokyo_4["oil"].interpolate()
tokyo_4["n225"] = tokyo_4["n225"].interpolate()
tokyo_4[tokyo_4.isnull().any(axis=1)]
| tokyo | oil | n225 | jp_unemp | jp_gdp | |
|---|---|---|---|---|---|
| Date |
# Take first difference
tokyo_4_diff = tokyo_4.diff().dropna()
# Split into train/test
train_diff4 = tokyo_4_diff[tokyo_4_diff.index.year!=2021]
test_diff4 = tokyo_4_diff[tokyo_4_diff.index.year==2021]
from statsmodels.tsa.vector_ar.var_model import VAR
var_model4 = VAR(train_diff4)
results_aic4 =[]
for p in range(1,21):
results4 = var_model4.fit(p)
results_aic4.append(results4.aic)
# Min=2
plt.plot(list(np.arange(1,21,1)), results_aic4, marker="o")
plt.xlabel("Order")
plt.ylabel("AIC")
plt.show()
var2_model4 = var_model4.fit(14)
print(var2_model4.summary())
Summary of Regression Results
==================================
Model: VAR
Method: OLS
Date: Wed, 01, Jun, 2022
Time: 11:42:13
--------------------------------------------------------------------
No. of Equations: 5.00000 BIC: 29.2735
Nobs: 960.000 HQIC: 28.1591
Log likelihood: -19643.3 FPE: 8.55638e+11
AIC: 27.4738 Det(Omega_mle): 5.98898e+11
--------------------------------------------------------------------
Results for equation tokyo
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 0.129350 1.274157 0.102 0.919
L1.tokyo 0.016725 0.036604 0.457 0.648
L1.oil -0.028633 0.681797 -0.042 0.967
L1.n225 -0.001105 0.006018 -0.184 0.854
L1.jp_unemp -24.229457 41.519050 -0.584 0.560
L1.jp_gdp 0.000133 0.000812 0.164 0.870
L2.tokyo -0.064869 0.036636 -1.771 0.077
L2.oil -0.166729 0.632665 -0.264 0.792
L2.n225 0.001822 0.006011 0.303 0.762
L2.jp_unemp 23.692796 41.522631 0.571 0.568
L2.jp_gdp -0.000444 0.000812 -0.547 0.584
L3.tokyo -0.100375 0.036697 -2.735 0.006
L3.oil -0.034243 0.629711 -0.054 0.957
L3.n225 0.005857 0.006025 0.972 0.331
L3.jp_unemp 17.504739 41.536050 0.421 0.673
L3.jp_gdp -0.000317 0.000813 -0.390 0.696
L4.tokyo -0.097246 0.037056 -2.624 0.009
L4.oil 0.724615 0.622571 1.164 0.244
L4.n225 0.002353 0.006005 0.392 0.695
L4.jp_unemp -21.802086 41.532733 -0.525 0.600
L4.jp_gdp -0.000432 0.000808 -0.535 0.593
L5.tokyo 0.102288 0.037213 2.749 0.006
L5.oil 1.270442 0.622807 2.040 0.041
L5.n225 -0.006042 0.006038 -1.001 0.317
L5.jp_unemp 37.396623 41.507433 0.901 0.368
L5.jp_gdp -0.000901 0.000806 -1.119 0.263
L6.tokyo -0.068382 0.037412 -1.828 0.068
L6.oil 0.113524 0.618843 0.183 0.854
L6.n225 0.005998 0.006059 0.990 0.322
L6.jp_unemp -36.427521 41.432628 -0.879 0.379
L6.jp_gdp 0.000494 0.000803 0.615 0.539
L7.tokyo 0.023498 0.037564 0.626 0.532
L7.oil 0.729607 0.618033 1.181 0.238
L7.n225 -0.008081 0.006181 -1.307 0.191
L7.jp_unemp -9.059245 41.379854 -0.219 0.827
L7.jp_gdp -0.000706 0.000800 -0.882 0.378
L8.tokyo -0.044606 0.037493 -1.190 0.234
L8.oil -0.440661 0.621036 -0.710 0.478
L8.n225 -0.012703 0.006157 -2.063 0.039
L8.jp_unemp 12.313401 41.334176 0.298 0.766
L8.jp_gdp 0.000970 0.000798 1.215 0.225
L9.tokyo -0.026675 0.037550 -0.710 0.477
L9.oil 0.270166 0.622797 0.434 0.664
L9.n225 -0.011323 0.006185 -1.831 0.067
L9.jp_unemp -11.739830 41.321732 -0.284 0.776
L9.jp_gdp -0.000996 0.000787 -1.266 0.206
L10.tokyo 0.004736 0.037470 0.126 0.899
L10.oil 0.264198 0.621966 0.425 0.671
L10.n225 -0.005669 0.006173 -0.918 0.358
L10.jp_unemp -10.212822 41.318051 -0.247 0.805
L10.jp_gdp -0.000348 0.000776 -0.448 0.654
L11.tokyo 0.068734 0.037471 1.834 0.067
L11.oil 0.074409 0.621755 0.120 0.905
L11.n225 -0.013184 0.006130 -2.151 0.032
L11.jp_unemp 24.897222 41.300049 0.603 0.547
L11.jp_gdp -0.000003 0.000773 -0.004 0.997
L12.tokyo -0.044714 0.037785 -1.183 0.237
L12.oil -0.630755 0.617852 -1.021 0.307
L12.n225 0.004000 0.006187 0.646 0.518
L12.jp_unemp -0.336486 41.272195 -0.008 0.993
L12.jp_gdp -0.000149 0.000763 -0.195 0.845
L13.tokyo 0.004004 0.037722 0.106 0.915
L13.oil -0.634105 0.603397 -1.051 0.293
L13.n225 -0.001136 0.006118 -0.186 0.853
L13.jp_unemp 13.934867 41.276969 0.338 0.736
L13.jp_gdp -0.000178 0.000768 -0.232 0.817
L14.tokyo 0.046033 0.037786 1.218 0.223
L14.oil -0.058939 0.551484 -0.107 0.915
L14.n225 -0.015614 0.006122 -2.550 0.011
L14.jp_unemp 7.741784 41.634133 0.186 0.852
L14.jp_gdp -0.000784 0.000947 -0.827 0.408
===============================================================================
Results for equation oil
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const -0.052044 0.059769 -0.871 0.384
L1.tokyo -0.000988 0.001717 -0.576 0.565
L1.oil -0.240821 0.031982 -7.530 0.000
L1.n225 -0.000111 0.000282 -0.392 0.695
L1.jp_unemp 0.769826 1.947607 0.395 0.693
L1.jp_gdp -0.000057 0.000038 -1.487 0.137
L2.tokyo 0.001465 0.001719 0.852 0.394
L2.oil -0.157049 0.029678 -5.292 0.000
L2.n225 0.000646 0.000282 2.290 0.022
L2.jp_unemp 0.968968 1.947775 0.497 0.619
L2.jp_gdp -0.000056 0.000038 -1.482 0.138
L3.tokyo -0.003142 0.001721 -1.825 0.068
L3.oil -0.099236 0.029539 -3.359 0.001
L3.n225 0.000310 0.000283 1.098 0.272
L3.jp_unemp -1.003686 1.948405 -0.515 0.606
L3.jp_gdp 0.000002 0.000038 0.045 0.964
L4.tokyo -0.000407 0.001738 -0.234 0.815
L4.oil -0.070118 0.029204 -2.401 0.016
L4.n225 0.000518 0.000282 1.838 0.066
L4.jp_unemp -0.504780 1.948249 -0.259 0.796
L4.jp_gdp 0.000045 0.000038 1.176 0.240
L5.tokyo -0.002722 0.001746 -1.559 0.119
L5.oil 0.012806 0.029215 0.438 0.661
L5.n225 0.000586 0.000283 2.068 0.039
L5.jp_unemp -0.924974 1.947062 -0.475 0.635
L5.jp_gdp 0.000018 0.000038 0.464 0.643
L6.tokyo -0.001736 0.001755 -0.989 0.322
L6.oil 0.024871 0.029029 0.857 0.392
L6.n225 0.000637 0.000284 2.241 0.025
L6.jp_unemp -0.626759 1.943553 -0.322 0.747
L6.jp_gdp 0.000028 0.000038 0.748 0.455
L7.tokyo 0.000804 0.001762 0.456 0.648
L7.oil -0.015108 0.028991 -0.521 0.602
L7.n225 0.000388 0.000290 1.339 0.181
L7.jp_unemp -0.665283 1.941078 -0.343 0.732
L7.jp_gdp 0.000062 0.000038 1.648 0.099
L8.tokyo -0.001468 0.001759 -0.835 0.404
L8.oil -0.023386 0.029132 -0.803 0.422
L8.n225 0.000547 0.000289 1.895 0.058
L8.jp_unemp 0.366181 1.938935 0.189 0.850
L8.jp_gdp -0.000006 0.000037 -0.162 0.871
L9.tokyo 0.001288 0.001761 0.731 0.465
L9.oil -0.025044 0.029215 -0.857 0.391
L9.n225 0.000115 0.000290 0.398 0.691
L9.jp_unemp 0.557026 1.938351 0.287 0.774
L9.jp_gdp 0.000045 0.000037 1.230 0.219
L10.tokyo 0.001121 0.001758 0.638 0.524
L10.oil -0.047114 0.029176 -1.615 0.106
L10.n225 0.000033 0.000290 0.115 0.908
L10.jp_unemp 1.264083 1.938179 0.652 0.514
L10.jp_gdp 0.000025 0.000036 0.687 0.492
L11.tokyo -0.005609 0.001758 -3.191 0.001
L11.oil -0.065137 0.029166 -2.233 0.026
L11.n225 0.000762 0.000288 2.651 0.008
L11.jp_unemp 0.805814 1.937334 0.416 0.677
L11.jp_gdp 0.000008 0.000036 0.210 0.834
L12.tokyo 0.000260 0.001772 0.147 0.883
L12.oil -0.057158 0.028983 -1.972 0.049
L12.n225 0.000379 0.000290 1.306 0.192
L12.jp_unemp -1.542192 1.936028 -0.797 0.426
L12.jp_gdp 0.000018 0.000036 0.492 0.623
L13.tokyo 0.000065 0.001769 0.037 0.971
L13.oil 0.001391 0.028305 0.049 0.961
L13.n225 0.000627 0.000287 2.183 0.029
L13.jp_unemp -7.709431 1.936252 -3.982 0.000
L13.jp_gdp 0.000819 0.000036 22.733 0.000
L14.tokyo -0.000391 0.001772 -0.221 0.825
L14.oil -0.021923 0.025869 -0.847 0.397
L14.n225 -0.000162 0.000287 -0.563 0.573
L14.jp_unemp 2.469414 1.953006 1.264 0.206
L14.jp_gdp -0.000471 0.000044 -10.608 0.000
===============================================================================
Results for equation n225
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 9.490934 7.853279 1.209 0.227
L1.tokyo 0.035177 0.225609 0.156 0.876
L1.oil 13.358701 4.202263 3.179 0.001
L1.n225 0.002026 0.037092 0.055 0.956
L1.jp_unemp 15.889881 255.903108 0.062 0.950
L1.jp_gdp 0.004039 0.005003 0.807 0.419
L2.tokyo -0.360740 0.225807 -1.598 0.110
L2.oil 14.074785 3.899437 3.609 0.000
L2.n225 0.072467 0.037046 1.956 0.050
L2.jp_unemp -111.736583 255.925179 -0.437 0.662
L2.jp_gdp 0.004443 0.005007 0.887 0.375
L3.tokyo -0.559796 0.226180 -2.475 0.013
L3.oil 8.233740 3.881231 2.121 0.034
L3.n225 0.033962 0.037132 0.915 0.360
L3.jp_unemp 302.529739 256.007886 1.182 0.237
L3.jp_gdp -0.006424 0.005012 -1.282 0.200
L4.tokyo -0.083340 0.228392 -0.365 0.715
L4.oil 7.703406 3.837225 2.008 0.045
L4.n225 0.048352 0.037010 1.306 0.191
L4.jp_unemp 86.194576 255.987439 0.337 0.736
L4.jp_gdp -0.004920 0.004980 -0.988 0.323
L5.tokyo 0.288460 0.229362 1.258 0.209
L5.oil 1.330679 3.838677 0.347 0.729
L5.n225 -0.037900 0.037214 -1.018 0.308
L5.jp_unemp 72.241820 255.831503 0.282 0.778
L5.jp_gdp -0.009169 0.004966 -1.846 0.065
L6.tokyo -0.228389 0.230589 -0.990 0.322
L6.oil 1.133284 3.814248 0.297 0.766
L6.n225 -0.032775 0.037343 -0.878 0.380
L6.jp_unemp -308.924452 255.370443 -1.210 0.226
L6.jp_gdp 0.004558 0.004950 0.921 0.357
L7.tokyo 0.523991 0.231523 2.263 0.024
L7.oil -6.176508 3.809251 -1.621 0.105
L7.n225 -0.070185 0.038096 -1.842 0.065
L7.jp_unemp -167.253858 255.045171 -0.656 0.512
L7.jp_gdp -0.000452 0.004932 -0.092 0.927
L8.tokyo 0.013497 0.231090 0.058 0.953
L8.oil 6.312837 3.827764 1.649 0.099
L8.n225 -0.061450 0.037951 -1.619 0.105
L8.jp_unemp 1.217440 254.763635 0.005 0.996
L8.jp_gdp 0.011576 0.004920 2.353 0.019
L9.tokyo 0.251338 0.231438 1.086 0.277
L9.oil 6.763598 3.838614 1.762 0.078
L9.n225 0.000676 0.038119 0.018 0.986
L9.jp_unemp -130.776660 254.686933 -0.513 0.608
L9.jp_gdp -0.012822 0.004852 -2.643 0.008
L10.tokyo 0.008341 0.230948 0.036 0.971
L10.oil -3.100924 3.833493 -0.809 0.419
L10.n225 0.017709 0.038049 0.465 0.642
L10.jp_unemp -212.623498 254.664246 -0.835 0.404
L10.jp_gdp 0.000832 0.004784 0.174 0.862
L11.tokyo 0.544749 0.230951 2.359 0.018
L11.oil -6.670563 3.832195 -1.741 0.082
L11.n225 -0.053284 0.037784 -1.410 0.158
L11.jp_unemp 167.675546 254.553292 0.659 0.510
L11.jp_gdp 0.002887 0.004765 0.606 0.544
L12.tokyo -0.171480 0.232889 -0.736 0.462
L12.oil -0.957744 3.808139 -0.251 0.801
L12.n225 0.028547 0.038131 0.749 0.454
L12.jp_unemp -219.537466 254.381611 -0.863 0.388
L12.jp_gdp -0.013040 0.004705 -2.771 0.006
L13.tokyo 0.078293 0.232500 0.337 0.736
L13.oil 1.377585 3.719042 0.370 0.711
L13.n225 0.065810 0.037711 1.745 0.081
L13.jp_unemp 78.291379 254.411038 0.308 0.758
L13.jp_gdp 0.001290 0.004731 0.273 0.785
L14.tokyo 0.330204 0.232894 1.418 0.156
L14.oil 4.007392 3.399076 1.179 0.238
L14.n225 -0.046818 0.037733 -1.241 0.215
L14.jp_unemp 166.025174 256.612424 0.647 0.518
L14.jp_gdp -0.003510 0.005840 -0.601 0.548
===============================================================================
Results for equation jp_unemp
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 0.000295 0.001033 0.286 0.775
L1.tokyo 0.000002 0.000030 0.057 0.955
L1.oil 0.000574 0.000553 1.039 0.299
L1.n225 -0.000000 0.000005 -0.005 0.996
L1.jp_unemp -0.001068 0.033661 -0.032 0.975
L1.jp_gdp 0.000001 0.000001 0.878 0.380
L2.tokyo 0.000008 0.000030 0.262 0.793
L2.oil 0.000452 0.000513 0.882 0.378
L2.n225 -0.000006 0.000005 -1.165 0.244
L2.jp_unemp -0.003765 0.033664 -0.112 0.911
L2.jp_gdp 0.000000 0.000001 0.164 0.870
L3.tokyo 0.000004 0.000030 0.119 0.906
L3.oil 0.000312 0.000511 0.611 0.541
L3.n225 0.000005 0.000005 0.949 0.343
L3.jp_unemp 0.005519 0.033675 0.164 0.870
L3.jp_gdp -0.000000 0.000001 -0.007 0.994
L4.tokyo 0.000032 0.000030 1.067 0.286
L4.oil 0.000124 0.000505 0.246 0.805
L4.n225 -0.000006 0.000005 -1.296 0.195
L4.jp_unemp -0.002105 0.033672 -0.063 0.950
L4.jp_gdp 0.000000 0.000001 0.041 0.968
L5.tokyo -0.000007 0.000030 -0.245 0.806
L5.oil 0.000160 0.000505 0.317 0.751
L5.n225 -0.000003 0.000005 -0.617 0.537
L5.jp_unemp -0.003412 0.033652 -0.101 0.919
L5.jp_gdp 0.000000 0.000001 0.027 0.979
L6.tokyo 0.000010 0.000030 0.323 0.747
L6.oil 0.000015 0.000502 0.031 0.975
L6.n225 0.000002 0.000005 0.353 0.724
L6.jp_unemp -0.004347 0.033591 -0.129 0.897
L6.jp_gdp 0.000000 0.000001 0.717 0.473
L7.tokyo -0.000036 0.000030 -1.193 0.233
L7.oil 0.000154 0.000501 0.306 0.759
L7.n225 0.000002 0.000005 0.358 0.721
L7.jp_unemp 0.002203 0.033548 0.066 0.948
L7.jp_gdp -0.000000 0.000001 -0.476 0.634
L8.tokyo -0.000009 0.000030 -0.297 0.766
L8.oil -0.001016 0.000504 -2.017 0.044
L8.n225 -0.000002 0.000005 -0.347 0.729
L8.jp_unemp -0.003088 0.033511 -0.092 0.927
L8.jp_gdp -0.000000 0.000001 -0.212 0.832
L9.tokyo 0.000040 0.000030 1.326 0.185
L9.oil -0.000744 0.000505 -1.474 0.140
L9.n225 -0.000003 0.000005 -0.547 0.584
L9.jp_unemp 0.002999 0.033501 0.090 0.929
L9.jp_gdp -0.000000 0.000001 -0.681 0.496
L10.tokyo 0.000020 0.000030 0.662 0.508
L10.oil -0.000211 0.000504 -0.417 0.676
L10.n225 -0.000004 0.000005 -0.796 0.426
L10.jp_unemp -0.000975 0.033498 -0.029 0.977
L10.jp_gdp 0.000000 0.000001 0.067 0.947
L11.tokyo 0.000012 0.000030 0.402 0.687
L11.oil -0.000256 0.000504 -0.509 0.611
L11.n225 0.000005 0.000005 0.952 0.341
L11.jp_unemp 0.000898 0.033484 0.027 0.979
L11.jp_gdp -0.000000 0.000001 -0.283 0.777
L12.tokyo -0.000018 0.000031 -0.597 0.551
L12.oil 0.000068 0.000501 0.135 0.892
L12.n225 -0.000000 0.000005 -0.046 0.963
L12.jp_unemp -0.005531 0.033461 -0.165 0.869
L12.jp_gdp 0.000000 0.000001 0.094 0.925
L13.tokyo 0.000018 0.000031 0.577 0.564
L13.oil -0.000156 0.000489 -0.318 0.751
L13.n225 -0.000003 0.000005 -0.659 0.510
L13.jp_unemp -0.000186 0.033465 -0.006 0.996
L13.jp_gdp 0.000000 0.000001 0.073 0.942
L14.tokyo -0.000016 0.000031 -0.511 0.610
L14.oil 0.000051 0.000447 0.114 0.909
L14.n225 -0.000008 0.000005 -1.586 0.113
L14.jp_unemp -0.008884 0.033755 -0.263 0.792
L14.jp_gdp -0.000001 0.000001 -0.656 0.512
===============================================================================
Results for equation jp_gdp
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 3.945648 52.752526 0.075 0.940
L1.tokyo 2.901616 1.515475 1.915 0.056
L1.oil -15.965381 28.227698 -0.566 0.572
L1.n225 -0.007127 0.249159 -0.029 0.977
L1.jp_unemp 278.809583 1718.968052 0.162 0.871
L1.jp_gdp -0.018299 0.033605 -0.545 0.586
L2.tokyo -0.696877 1.516803 -0.459 0.646
L2.oil 18.943446 26.193536 0.723 0.470
L2.n225 -0.014815 0.248849 -0.060 0.953
L2.jp_unemp -154.631785 1719.116307 -0.090 0.928
L2.jp_gdp -0.012458 0.033631 -0.370 0.711
L3.tokyo -5.658004 1.519310 -3.724 0.000
L3.oil 47.756189 26.071242 1.832 0.067
L3.n225 0.112994 0.249427 0.453 0.651
L3.jp_unemp -724.344314 1719.671868 -0.421 0.674
L3.jp_gdp 0.013762 0.033667 0.409 0.683
L4.tokyo 1.801590 1.534169 1.174 0.240
L4.oil 32.903868 25.775641 1.277 0.202
L4.n225 0.556146 0.248608 2.237 0.025
L4.jp_unemp 439.752071 1719.534520 0.256 0.798
L4.jp_gdp 0.028081 0.033450 0.839 0.401
L5.tokyo -4.275871 1.540684 -2.775 0.006
L5.oil 7.687681 25.785396 0.298 0.766
L5.n225 -1.001938 0.249976 -4.008 0.000
L5.jp_unemp -567.038507 1718.487058 -0.330 0.741
L5.jp_gdp -0.003510 0.033361 -0.105 0.916
L6.tokyo 6.041982 1.548923 3.901 0.000
L6.oil 20.777342 25.621299 0.811 0.417
L6.n225 -1.443493 0.250841 -5.755 0.000
L6.jp_unemp -192.978928 1715.390000 -0.112 0.910
L6.jp_gdp 0.033333 0.033249 1.003 0.316
L7.tokyo 0.988582 1.555202 0.636 0.525
L7.oil 60.062806 25.587733 2.347 0.019
L7.n225 -0.606901 0.255902 -2.372 0.018
L7.jp_unemp 21.416850 1713.205062 0.013 0.990
L7.jp_gdp 0.010253 0.033133 0.309 0.757
L8.tokyo -4.050084 1.552290 -2.609 0.009
L8.oil 29.208217 25.712089 1.136 0.256
L8.n225 0.163150 0.254929 0.640 0.522
L8.jp_unemp -12.190116 1711.313912 -0.007 0.994
L8.jp_gdp 0.025064 0.033048 0.758 0.448
L9.tokyo -1.512220 1.554631 -0.973 0.331
L9.oil 69.581093 25.784974 2.699 0.007
L9.n225 0.257717 0.256055 1.006 0.314
L9.jp_unemp 139.217509 1710.798685 0.081 0.935
L9.jp_gdp -0.007602 0.032589 -0.233 0.816
L10.tokyo -2.889163 1.551337 -1.862 0.063
L10.oil 55.784028 25.750575 2.166 0.030
L10.n225 0.086263 0.255587 0.338 0.736
L10.jp_unemp 665.001732 1710.646288 0.389 0.697
L10.jp_gdp -0.016064 0.032133 -0.500 0.617
L11.tokyo -2.020448 1.551358 -1.302 0.193
L11.oil 22.373141 25.741855 0.869 0.385
L11.n225 0.826177 0.253803 3.255 0.001
L11.jp_unemp 363.143270 1709.900980 0.212 0.832
L11.jp_gdp -0.017042 0.032005 -0.532 0.594
L12.tokyo 1.318477 1.564378 0.843 0.399
L12.oil -0.706699 25.580262 -0.028 0.978
L12.n225 0.204338 0.256138 0.798 0.425
L12.jp_unemp -139.758907 1708.747753 -0.082 0.935
L12.jp_gdp -0.037676 0.031606 -1.192 0.233
L13.tokyo 2.182687 1.561763 1.398 0.162
L13.oil -16.257410 24.981775 -0.651 0.515
L13.n225 0.206257 0.253313 0.814 0.416
L13.jp_unemp 476.176932 1708.945425 0.279 0.781
L13.jp_gdp 0.008251 0.031779 0.260 0.795
L14.tokyo -1.883682 1.564412 -1.204 0.229
L14.oil 4.666174 22.832482 0.204 0.838
L14.n225 -0.149162 0.253459 -0.589 0.556
L14.jp_unemp -885.648326 1723.732707 -0.514 0.607
L14.jp_gdp 0.003288 0.039226 0.084 0.933
===============================================================================
Correlation matrix of residuals
tokyo oil n225 jp_unemp jp_gdp
tokyo 1.000000 0.063808 0.404194 -0.084183 0.010154
oil 0.063808 1.000000 0.161064 -0.007922 -0.000862
n225 0.404194 0.161064 1.000000 -0.082268 0.063166
jp_unemp -0.084183 -0.007922 -0.082268 1.000000 0.047640
jp_gdp 0.010154 -0.000862 0.063166 0.047640 1.000000
# Calculate yhat
yhat_var2_4 = var2_model4.forecast(var_model4.endog, steps=len(test_diff4))
# clean up yhat
df_yhat_var2_4 = pd.DataFrame(yhat_var2_4)
df_yhat_var2_4.index = test_diff4.index
df_yhat_var2_4.columns = test_diff4.columns
df_yhat_var2_4
| tokyo | oil | n225 | jp_unemp | jp_gdp | |
|---|---|---|---|---|---|
| Date | |||||
| 2021-01-04 | 6.479636 | 0.102098 | 86.754079 | 0.002943 | -506.777595 |
| 2021-01-05 | -1.216366 | 0.075738 | -12.227668 | 0.005096 | -21.508014 |
| 2021-01-06 | 3.386518 | -0.012620 | 43.704692 | -0.005312 | -140.902446 |
| 2021-01-07 | -9.955103 | 0.331145 | -24.920802 | -0.001603 | -868.136344 |
| 2021-01-08 | 2.861119 | 0.558415 | 13.154791 | -0.000842 | -847.139945 |
| ... | ... | ... | ... | ... | ... |
| 2021-11-29 | -0.300903 | -0.001475 | 8.935765 | 0.000080 | -1.497658 |
| 2021-11-30 | -0.300903 | -0.001475 | 8.935766 | 0.000080 | -1.497658 |
| 2021-12-01 | -0.300903 | -0.001475 | 8.935766 | 0.000080 | -1.497661 |
| 2021-12-02 | -0.300903 | -0.001475 | 8.935766 | 0.000080 | -1.497661 |
| 2021-12-03 | -0.300903 | -0.001475 | 8.935765 | 0.000080 | -1.497659 |
226 rows × 5 columns
# Recover first order differencing
df_pred_var2_4 = pd.concat([tokyo_4[tokyo_4.index==pd.datetime(2020,12,30)], df_yhat_var2_4], axis=0).cumsum()
df_pred_var2_4 = df_pred_var2_4[df_pred_var2_4.index!=pd.datetime(2020,12,30)]
y_pred_var2_4 = df_pred_var2_4["tokyo"]
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_var2_4.plot(ax=ax, label="VAR(14) with Oil, N225, Unemployement, GDP", linestyle="dashed")
plt.legend()
plt.show()
var2_metrics4 = calc_metrics(y_test, y_pred_var2_4, return_dict=True)
Test Error Metrics MAE: 155.0108 MSE: 35544.422 MAPE: 0.0744 sMAPE: 0.0706 MASE: -6.6604
# Merge data and fill NAs with interpolation
tokyo_5 = pd.concat([tokyo, oil_s, n225_s, unemp_s, gdp_s, usdjpy_s], axis=1)
tokyo_5["jp_gdp"] = tokyo_5["jp_gdp"].fillna(method="ffill")
tokyo_5["jp_unemp"] = tokyo_5["jp_unemp"].fillna(method="ffill")
tokyo_5 = tokyo_5.dropna(subset=["tokyo"])
tokyo_5["oil"] = tokyo_5["oil"].interpolate()
tokyo_5["n225"] = tokyo_5["n225"].interpolate()
tokyo_5["usdjpy"] = tokyo_5["usdjpy"].interpolate()
tokyo_5[tokyo_5.isnull().any(axis=1)]
| tokyo | oil | n225 | jp_unemp | jp_gdp | usdjpy | |
|---|---|---|---|---|---|---|
| Date |
# Take first difference
tokyo_5_diff = tokyo_5.diff().dropna()
# Split into train/test
train_diff5 = tokyo_5_diff[tokyo_5_diff.index.year!=2021]
test_diff5 = tokyo_5_diff[tokyo_5_diff.index.year==2021]
from statsmodels.tsa.vector_ar.var_model import VAR
var_model5 = VAR(train_diff5)
results_aic5 =[]
for p in range(1,21):
results5 = var_model5.fit(p)
results_aic5.append(results5.aic)
# Min=2
plt.plot(list(np.arange(1,21,1)), results_aic5, marker="o")
plt.xlabel("Order")
plt.ylabel("AIC")
plt.show()
var2_model5 = var_model5.fit(14)
print(var2_model5.summary())
Summary of Regression Results
==================================
Model: VAR
Method: OLS
Date: Wed, 01, Jun, 2022
Time: 18:42:17
--------------------------------------------------------------------
No. of Equations: 6.00000 BIC: 28.6103
Nobs: 960.000 HQIC: 27.0094
Log likelihood: -20155.0 FPE: 2.01203e+11
AIC: 26.0248 Det(Omega_mle): 1.20938e+11
--------------------------------------------------------------------
Results for equation tokyo
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 0.462762 1.295484 0.357 0.721
L1.tokyo 0.022950 0.036923 0.622 0.534
L1.oil -0.007295 0.683806 -0.011 0.991
L1.n225 -0.004750 0.006469 -0.734 0.463
L1.jp_unemp -24.569380 41.744671 -0.589 0.556
L1.jp_gdp 0.000296 0.000838 0.353 0.724
L1.usdjpy 4.214655 2.785318 1.513 0.130
L2.tokyo -0.064317 0.036941 -1.741 0.082
L2.oil -0.031513 0.636530 -0.050 0.961
L2.n225 0.000124 0.006535 0.019 0.985
L2.jp_unemp 17.293702 41.732744 0.414 0.679
L2.jp_gdp -0.000199 0.000838 -0.237 0.813
L2.usdjpy -0.485347 2.823843 -0.172 0.864
L3.tokyo -0.105619 0.036990 -2.855 0.004
L3.oil 0.031918 0.636013 0.050 0.960
L3.n225 0.005300 0.006566 0.807 0.420
L3.jp_unemp 16.948100 41.738858 0.406 0.685
L3.jp_gdp -0.000352 0.000837 -0.421 0.674
L3.usdjpy 1.477607 2.824899 0.523 0.601
L4.tokyo -0.084541 0.037415 -2.260 0.024
L4.oil 0.889796 0.627807 1.417 0.156
L4.n225 -0.000916 0.006586 -0.139 0.889
L4.jp_unemp -12.608159 41.723232 -0.302 0.763
L4.jp_gdp -0.000584 0.000825 -0.707 0.479
L4.usdjpy 3.644835 2.827963 1.289 0.197
L5.tokyo 0.096493 0.037505 2.573 0.010
L5.oil 1.379520 0.627713 2.198 0.028
L5.n225 -0.003727 0.006631 -0.562 0.574
L5.jp_unemp 29.236720 41.786820 0.700 0.484
L5.jp_gdp -0.001074 0.000824 -1.304 0.192
L5.usdjpy -3.479371 2.813602 -1.237 0.216
L6.tokyo -0.069677 0.037558 -1.855 0.064
L6.oil 0.200759 0.623501 0.322 0.747
L6.n225 0.005802 0.006614 0.877 0.380
L6.jp_unemp -27.824564 41.796712 -0.666 0.506
L6.jp_gdp 0.000034 0.000822 0.041 0.967
L6.usdjpy 2.577373 2.806375 0.918 0.358
L7.tokyo 0.024201 0.037673 0.642 0.521
L7.oil 0.793055 0.622260 1.274 0.202
L7.n225 -0.006948 0.006720 -1.034 0.301
L7.jp_unemp -6.487092 41.692998 -0.156 0.876
L7.jp_gdp -0.001049 0.000815 -1.287 0.198
L7.usdjpy -2.516493 2.782723 -0.904 0.366
L8.tokyo -0.042823 0.037589 -1.139 0.255
L8.oil -0.417578 0.626188 -0.667 0.505
L8.n225 -0.014279 0.006686 -2.136 0.033
L8.jp_unemp -4.160973 41.664888 -0.100 0.920
L8.jp_gdp 0.001056 0.000810 1.304 0.192
L8.usdjpy 0.799687 2.789571 0.287 0.774
L9.tokyo -0.026800 0.037600 -0.713 0.476
L9.oil 0.297287 0.628155 0.473 0.636
L9.n225 -0.020012 0.006737 -2.970 0.003
L9.jp_unemp -8.968359 41.684022 -0.215 0.830
L9.jp_gdp -0.000988 0.000800 -1.235 0.217
L9.usdjpy 10.575145 2.817773 3.753 0.000
L10.tokyo 0.008431 0.037531 0.225 0.822
L10.oil 0.264688 0.624534 0.424 0.672
L10.n225 -0.010589 0.006746 -1.570 0.116
L10.jp_unemp -11.403659 41.717569 -0.273 0.785
L10.jp_gdp -0.000192 0.000790 -0.244 0.808
L10.usdjpy 1.032596 2.829702 0.365 0.715
L11.tokyo 0.068684 0.037497 1.832 0.067
L11.oil 0.184171 0.623367 0.295 0.768
L11.n225 -0.014514 0.006707 -2.164 0.030
L11.jp_unemp 19.063912 41.712303 0.457 0.648
L11.jp_gdp 0.000152 0.000786 0.194 0.846
L11.usdjpy 2.438795 2.833278 0.861 0.389
L12.tokyo -0.055958 0.037826 -1.479 0.139
L12.oil -0.491436 0.618963 -0.794 0.427
L12.n225 0.005143 0.006817 0.754 0.451
L12.jp_unemp -9.441946 41.696194 -0.226 0.821
L12.jp_gdp 0.000045 0.000772 0.058 0.953
L12.usdjpy 0.612623 2.849921 0.215 0.830
L13.tokyo 0.003381 0.037771 0.090 0.929
L13.oil -0.615713 0.605055 -1.018 0.309
L13.n225 0.001385 0.006736 0.206 0.837
L13.jp_unemp 7.863036 41.672508 0.189 0.850
L13.jp_gdp -0.000373 0.000772 -0.483 0.629
L13.usdjpy -2.481389 2.826686 -0.878 0.380
L14.tokyo 0.037451 0.037890 0.988 0.323
L14.oil 0.106331 0.554207 0.192 0.848
L14.n225 -0.012502 0.006650 -1.880 0.060
L14.jp_unemp 10.946918 41.826029 0.262 0.794
L14.jp_gdp -0.000966 0.000951 -1.015 0.310
L14.usdjpy -1.279820 2.767218 -0.462 0.644
===============================================================================
Results for equation oil
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const -0.072995 0.061202 -1.193 0.233
L1.tokyo -0.000795 0.001744 -0.456 0.649
L1.oil -0.244692 0.032305 -7.574 0.000
L1.n225 -0.000053 0.000306 -0.175 0.861
L1.jp_unemp 0.733254 1.972124 0.372 0.710
L1.jp_gdp -0.000045 0.000040 -1.128 0.259
L1.usdjpy -0.073698 0.131585 -0.560 0.575
L2.tokyo 0.001653 0.001745 0.947 0.344
L2.oil -0.160465 0.030071 -5.336 0.000
L2.n225 0.000818 0.000309 2.649 0.008
L2.jp_unemp 1.143342 1.971560 0.580 0.562
L2.jp_gdp -0.000056 0.000040 -1.421 0.155
L2.usdjpy -0.121798 0.133405 -0.913 0.361
L3.tokyo -0.003244 0.001747 -1.856 0.063
L3.oil -0.102085 0.030047 -3.398 0.001
L3.n225 0.000478 0.000310 1.542 0.123
L3.jp_unemp -0.546758 1.971849 -0.277 0.782
L3.jp_gdp -0.000009 0.000040 -0.216 0.829
L3.usdjpy -0.121726 0.133455 -0.912 0.362
L4.tokyo -0.000241 0.001768 -0.136 0.892
L4.oil -0.077465 0.029659 -2.612 0.009
L4.n225 0.000684 0.000311 2.198 0.028
L4.jp_unemp -0.337083 1.971111 -0.171 0.864
L4.jp_gdp 0.000041 0.000039 1.047 0.295
L4.usdjpy -0.183250 0.133600 -1.372 0.170
L5.tokyo -0.002437 0.001772 -1.375 0.169
L5.oil 0.004483 0.029655 0.151 0.880
L5.n225 0.000594 0.000313 1.896 0.058
L5.jp_unemp -0.976173 1.974115 -0.494 0.621
L5.jp_gdp 0.000026 0.000039 0.676 0.499
L5.usdjpy -0.025115 0.132922 -0.189 0.850
L6.tokyo -0.001822 0.001774 -1.027 0.305
L6.oil 0.014813 0.029456 0.503 0.615
L6.n225 0.000722 0.000312 2.309 0.021
L6.jp_unemp -0.229553 1.974582 -0.116 0.907
L6.jp_gdp 0.000025 0.000039 0.646 0.518
L6.usdjpy -0.113591 0.132580 -0.857 0.392
L7.tokyo 0.001210 0.001780 0.680 0.497
L7.oil -0.017045 0.029397 -0.580 0.562
L7.n225 0.000517 0.000317 1.630 0.103
L7.jp_unemp -0.352881 1.969683 -0.179 0.858
L7.jp_gdp 0.000062 0.000039 1.614 0.107
L7.usdjpy -0.174360 0.131463 -1.326 0.185
L8.tokyo -0.001284 0.001776 -0.723 0.470
L8.oil -0.025626 0.029583 -0.866 0.386
L8.n225 0.000566 0.000316 1.793 0.073
L8.jp_unemp 0.280090 1.968355 0.142 0.887
L8.jp_gdp 0.000005 0.000038 0.119 0.905
L8.usdjpy -0.008993 0.131786 -0.068 0.946
L9.tokyo 0.001257 0.001776 0.707 0.479
L9.oil -0.030816 0.029676 -1.038 0.299
L9.n225 0.000028 0.000318 0.089 0.929
L9.jp_unemp 0.536740 1.969259 0.273 0.785
L9.jp_gdp 0.000052 0.000038 1.389 0.165
L9.usdjpy 0.092555 0.133119 0.695 0.487
L10.tokyo 0.001404 0.001773 0.792 0.428
L10.oil -0.047731 0.029505 -1.618 0.106
L10.n225 0.000113 0.000319 0.355 0.723
L10.jp_unemp 1.943267 1.970843 0.986 0.324
L10.jp_gdp 0.000018 0.000037 0.474 0.635
L10.usdjpy -0.100987 0.133682 -0.755 0.450
L11.tokyo -0.005621 0.001771 -3.173 0.002
L11.oil -0.062407 0.029449 -2.119 0.034
L11.n225 0.001033 0.000317 3.261 0.001
L11.jp_unemp 0.826217 1.970595 0.419 0.675
L11.jp_gdp 0.000012 0.000037 0.336 0.737
L11.usdjpy -0.279507 0.133851 -2.088 0.037
L12.tokyo 0.000102 0.001787 0.057 0.954
L12.oil -0.060194 0.029241 -2.059 0.040
L12.n225 0.000402 0.000322 1.248 0.212
L12.jp_unemp -1.813176 1.969834 -0.920 0.357
L12.jp_gdp 0.000020 0.000036 0.546 0.585
L12.usdjpy 0.043315 0.134637 0.322 0.748
L13.tokyo 0.000101 0.001784 0.057 0.955
L13.oil 0.001514 0.028584 0.053 0.958
L13.n225 0.000574 0.000318 1.803 0.071
L13.jp_unemp -7.702875 1.968715 -3.913 0.000
L13.jp_gdp 0.000818 0.000036 22.408 0.000
L13.usdjpy -0.006521 0.133540 -0.049 0.961
L14.tokyo -0.000151 0.001790 -0.085 0.933
L14.oil -0.018899 0.026182 -0.722 0.470
L14.n225 -0.000147 0.000314 -0.469 0.639
L14.jp_unemp 2.995692 1.975967 1.516 0.130
L14.jp_gdp -0.000470 0.000045 -10.454 0.000
L14.usdjpy -0.021510 0.130730 -0.165 0.869
===============================================================================
Results for equation n225
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 10.738970 8.002447 1.342 0.180
L1.tokyo 0.061761 0.228078 0.271 0.787
L1.oil 13.849053 4.223994 3.279 0.001
L1.n225 -0.032734 0.039958 -0.819 0.413
L1.jp_unemp -21.336992 257.864568 -0.083 0.934
L1.jp_gdp 0.004565 0.005178 0.882 0.378
L1.usdjpy 39.529786 17.205423 2.298 0.022
L2.tokyo -0.340988 0.228191 -1.494 0.135
L2.oil 15.299042 3.931962 3.891 0.000
L2.n225 0.046154 0.040371 1.143 0.253
L2.jp_unemp -112.209789 257.790892 -0.435 0.663
L2.jp_gdp 0.004196 0.005176 0.811 0.418
L2.usdjpy 17.289427 17.443401 0.991 0.322
L3.tokyo -0.576113 0.228493 -2.521 0.012
L3.oil 8.939995 3.928769 2.276 0.023
L3.n225 0.030689 0.040557 0.757 0.449
L3.jp_unemp 327.334916 257.828656 1.270 0.204
L3.jp_gdp -0.009071 0.005170 -1.754 0.079
L3.usdjpy 9.082241 17.449923 0.520 0.603
L4.tokyo -0.070402 0.231118 -0.305 0.761
L4.oil 8.304019 3.878079 2.141 0.032
L4.n225 0.032117 0.040684 0.789 0.430
L4.jp_unemp 37.779147 257.732136 0.147 0.883
L4.jp_gdp -0.003906 0.005099 -0.766 0.444
L4.usdjpy 16.009440 17.468852 0.916 0.359
L5.tokyo 0.278928 0.231674 1.204 0.229
L5.oil 1.573213 3.877497 0.406 0.685
L5.n225 -0.035467 0.040960 -0.866 0.387
L5.jp_unemp 32.214267 258.124928 0.125 0.901
L5.jp_gdp -0.010327 0.005087 -2.030 0.042
L5.usdjpy 8.026272 17.380143 0.462 0.644
L6.tokyo -0.267908 0.232005 -1.155 0.248
L6.oil 1.922333 3.851479 0.499 0.618
L6.n225 -0.021587 0.040858 -0.528 0.597
L6.jp_unemp -268.159001 258.186032 -1.039 0.299
L6.jp_gdp 0.002084 0.005076 0.410 0.681
L6.usdjpy -4.017911 17.335497 -0.232 0.817
L7.tokyo 0.521061 0.232711 2.239 0.025
L7.oil -5.974025 3.843817 -1.554 0.120
L7.n225 -0.062365 0.041510 -1.502 0.133
L7.jp_unemp -139.540626 257.545374 -0.542 0.588
L7.jp_gdp -0.002175 0.005034 -0.432 0.666
L7.usdjpy 0.400842 17.189395 0.023 0.981
L8.tokyo 0.000855 0.232196 0.004 0.997
L8.oil 5.420432 3.868078 1.401 0.161
L8.n225 -0.084229 0.041303 -2.039 0.041
L8.jp_unemp -108.884864 257.371731 -0.423 0.672
L8.jp_gdp 0.013536 0.005002 2.706 0.007
L8.usdjpy 12.111315 17.231699 0.703 0.482
L9.tokyo 0.245269 0.232263 1.056 0.291
L9.oil 6.271595 3.880231 1.616 0.106
L9.n225 -0.027861 0.041617 -0.669 0.503
L9.jp_unemp -135.435012 257.489927 -0.526 0.599
L9.jp_gdp -0.012814 0.004941 -2.593 0.010
L9.usdjpy 33.808555 17.405907 1.942 0.052
L10.tokyo 0.019107 0.231837 0.082 0.934
L10.oil -2.815652 3.857861 -0.730 0.465
L10.n225 0.023609 0.041668 0.567 0.571
L10.jp_unemp -162.581773 257.697151 -0.631 0.528
L10.jp_gdp 0.000457 0.004881 0.094 0.925
L10.usdjpy -20.225220 17.479595 -1.157 0.247
L11.tokyo 0.551521 0.231625 2.381 0.017
L11.oil -6.464373 3.850652 -1.679 0.093
L11.n225 -0.045521 0.041430 -1.099 0.272
L11.jp_unemp 125.053411 257.664620 0.485 0.627
L11.jp_gdp 0.004126 0.004856 0.850 0.396
L11.usdjpy 0.999769 17.501686 0.057 0.954
L12.tokyo -0.231117 0.233657 -0.989 0.323
L12.oil -1.197107 3.823446 -0.313 0.754
L12.n225 0.022887 0.042112 0.543 0.587
L12.jp_unemp -278.822703 257.565112 -1.083 0.279
L12.jp_gdp -0.011449 0.004768 -2.401 0.016
L12.usdjpy 9.760300 17.604489 0.554 0.579
L13.tokyo 0.112013 0.233317 0.480 0.631
L13.oil 1.809845 3.737538 0.484 0.628
L13.n225 0.073661 0.041612 1.770 0.077
L13.jp_unemp 134.964531 257.418799 0.524 0.600
L13.jp_gdp 0.000010 0.004771 0.002 0.998
L13.usdjpy -0.077492 17.460964 -0.004 0.996
L14.tokyo 0.296407 0.234056 1.266 0.205
L14.oil 4.228412 3.423439 1.235 0.217
L14.n225 -0.008343 0.041081 -0.203 0.839
L14.jp_unemp 227.460506 258.367130 0.880 0.379
L14.jp_gdp -0.005190 0.005877 -0.883 0.377
L14.usdjpy -41.240057 17.093618 -2.413 0.016
===============================================================================
Results for equation jp_unemp
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 0.000538 0.001056 0.509 0.610
L1.tokyo 0.000004 0.000030 0.119 0.905
L1.oil 0.000533 0.000557 0.957 0.338
L1.n225 -0.000004 0.000005 -0.773 0.439
L1.jp_unemp 0.000422 0.034013 0.012 0.990
L1.jp_gdp 0.000001 0.000001 1.004 0.315
L1.usdjpy 0.005664 0.002269 2.496 0.013
L2.tokyo 0.000006 0.000030 0.193 0.847
L2.oil 0.000477 0.000519 0.919 0.358
L2.n225 -0.000006 0.000005 -1.158 0.247
L2.jp_unemp -0.001750 0.034004 -0.051 0.959
L2.jp_gdp 0.000000 0.000001 0.261 0.794
L2.usdjpy -0.001272 0.002301 -0.553 0.580
L3.tokyo 0.000002 0.000030 0.057 0.954
L3.oil 0.000425 0.000518 0.820 0.412
L3.n225 0.000006 0.000005 1.070 0.285
L3.jp_unemp -0.001362 0.034009 -0.040 0.968
L3.jp_gdp 0.000000 0.000001 0.379 0.704
L3.usdjpy -0.000446 0.002302 -0.194 0.846
L4.tokyo 0.000028 0.000030 0.917 0.359
L4.oil 0.000156 0.000512 0.305 0.760
L4.n225 -0.000007 0.000005 -1.235 0.217
L4.jp_unemp -0.004197 0.033996 -0.123 0.902
L4.jp_gdp 0.000000 0.000001 0.069 0.945
L4.usdjpy 0.001878 0.002304 0.815 0.415
L5.tokyo -0.000014 0.000031 -0.442 0.658
L5.oil 0.000193 0.000511 0.378 0.705
L5.n225 -0.000001 0.000005 -0.248 0.804
L5.jp_unemp -0.003547 0.034048 -0.104 0.917
L5.jp_gdp -0.000000 0.000001 -0.225 0.822
L5.usdjpy -0.002035 0.002293 -0.888 0.375
L6.tokyo 0.000013 0.000031 0.440 0.660
L6.oil 0.000078 0.000508 0.153 0.878
L6.n225 0.000004 0.000005 0.695 0.487
L6.jp_unemp 0.001988 0.034056 0.058 0.953
L6.jp_gdp 0.000000 0.000001 0.372 0.710
L6.usdjpy -0.001954 0.002287 -0.855 0.393
L7.tokyo -0.000035 0.000031 -1.156 0.248
L7.oil 0.000159 0.000507 0.314 0.754
L7.n225 0.000000 0.000005 0.056 0.955
L7.jp_unemp -0.001502 0.033971 -0.044 0.965
L7.jp_gdp -0.000000 0.000001 -0.343 0.732
L7.usdjpy 0.001546 0.002267 0.682 0.495
L8.tokyo -0.000011 0.000031 -0.347 0.729
L8.oil -0.001120 0.000510 -2.195 0.028
L8.n225 -0.000005 0.000005 -0.855 0.393
L8.jp_unemp -0.006484 0.033948 -0.191 0.849
L8.jp_gdp -0.000000 0.000001 -0.336 0.737
L8.usdjpy 0.001447 0.002273 0.637 0.524
L9.tokyo 0.000049 0.000031 1.588 0.112
L9.oil -0.000562 0.000512 -1.099 0.272
L9.n225 -0.000006 0.000005 -1.013 0.311
L9.jp_unemp 0.006847 0.033964 0.202 0.840
L9.jp_gdp -0.000000 0.000001 -0.728 0.467
L9.usdjpy 0.002906 0.002296 1.266 0.206
L10.tokyo 0.000019 0.000031 0.628 0.530
L10.oil -0.000191 0.000509 -0.376 0.707
L10.n225 -0.000005 0.000005 -0.997 0.319
L10.jp_unemp 0.000367 0.033991 0.011 0.991
L10.jp_gdp 0.000000 0.000001 0.104 0.917
L10.usdjpy 0.000764 0.002306 0.331 0.740
L11.tokyo 0.000010 0.000031 0.323 0.746
L11.oil -0.000232 0.000508 -0.457 0.647
L11.n225 0.000002 0.000005 0.437 0.662
L11.jp_unemp -0.008212 0.033987 -0.242 0.809
L11.jp_gdp -0.000000 0.000001 -0.289 0.773
L11.usdjpy 0.002479 0.002309 1.074 0.283
L12.tokyo -0.000021 0.000031 -0.672 0.502
L12.oil 0.000199 0.000504 0.394 0.694
L12.n225 0.000000 0.000006 0.076 0.939
L12.jp_unemp -0.005556 0.033974 -0.164 0.870
L12.jp_gdp 0.000000 0.000001 0.104 0.917
L12.usdjpy 0.000321 0.002322 0.138 0.890
L13.tokyo 0.000017 0.000031 0.558 0.577
L13.oil -0.000090 0.000493 -0.182 0.855
L13.n225 -0.000002 0.000005 -0.369 0.712
L13.jp_unemp -0.003413 0.033955 -0.101 0.920
L13.jp_gdp -0.000000 0.000001 -0.017 0.986
L13.usdjpy -0.000799 0.002303 -0.347 0.729
L14.tokyo -0.000023 0.000031 -0.748 0.454
L14.oil 0.000018 0.000452 0.040 0.968
L14.n225 -0.000010 0.000005 -1.776 0.076
L14.jp_unemp -0.012243 0.034080 -0.359 0.719
L14.jp_gdp -0.000001 0.000001 -0.680 0.496
L14.usdjpy 0.002463 0.002255 1.092 0.275
===============================================================================
Results for equation jp_gdp
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const -27.057165 52.259649 -0.518 0.605
L1.tokyo 2.962936 1.489451 1.989 0.047
L1.oil -15.546829 27.584616 -0.564 0.573
L1.n225 0.024418 0.260943 0.094 0.925
L1.jp_unemp 10.654805 1683.973919 0.006 0.995
L1.jp_gdp -0.038952 0.033812 -1.152 0.249
L1.usdjpy -169.784876 112.359306 -1.511 0.131
L2.tokyo 0.040784 1.490190 0.027 0.978
L2.oil 12.888738 25.677516 0.502 0.616
L2.n225 -0.194535 0.263639 -0.738 0.461
L2.jp_unemp -284.247643 1683.492776 -0.169 0.866
L2.jp_gdp -0.027332 0.033804 -0.809 0.419
L2.usdjpy 269.376209 113.913412 2.365 0.018
L3.tokyo -5.143803 1.492163 -3.447 0.001
L3.oil 40.216057 25.656662 1.567 0.117
L3.n225 -0.113964 0.264857 -0.430 0.667
L3.jp_unemp -269.516801 1683.739395 -0.160 0.873
L3.jp_gdp -0.011941 0.033764 -0.354 0.724
L3.usdjpy 206.319516 113.956003 1.811 0.070
L4.tokyo 1.852296 1.509307 1.227 0.220
L4.oil 29.887224 25.325633 1.180 0.238
L4.n225 0.548712 0.265687 2.065 0.039
L4.jp_unemp 700.400878 1683.109075 0.416 0.677
L4.jp_gdp 0.020001 0.033298 0.601 0.548
L4.usdjpy -7.989087 114.079619 -0.070 0.944
L5.tokyo -3.760420 1.512939 -2.486 0.013
L5.oil 4.808553 25.321835 0.190 0.849
L5.n225 -1.012478 0.267488 -3.785 0.000
L5.jp_unemp -1035.359735 1685.674184 -0.614 0.539
L5.jp_gdp 0.013122 0.033223 0.395 0.693
L5.usdjpy 131.827783 113.500304 1.161 0.245
L6.tokyo 5.129634 1.515097 3.386 0.001
L6.oil 11.095763 25.151925 0.441 0.659
L6.n225 -1.412947 0.266820 -5.296 0.000
L6.jp_unemp -706.967467 1686.073223 -0.419 0.675
L6.jp_gdp 0.046131 0.033150 1.392 0.164
L6.usdjpy 126.330127 113.208750 1.116 0.264
L7.tokyo 0.521069 1.519709 0.343 0.732
L7.oil 49.068629 25.101891 1.955 0.051
L7.n225 -0.246469 0.271082 -0.909 0.363
L7.jp_unemp 388.504703 1681.889431 0.231 0.817
L7.jp_gdp 0.002133 0.032877 0.065 0.948
L7.usdjpy -319.093216 112.254631 -2.843 0.004
L8.tokyo -3.683877 1.516346 -2.429 0.015
L8.oil 25.664734 25.260325 1.016 0.310
L8.n225 0.515541 0.269726 1.911 0.056
L8.jp_unemp 187.806143 1680.755462 0.112 0.911
L8.jp_gdp 0.039789 0.032663 1.218 0.223
L8.usdjpy -249.634684 112.530898 -2.218 0.027
L9.tokyo -2.307202 1.516786 -1.521 0.128
L9.oil 42.677158 25.339689 1.684 0.092
L9.n225 0.504498 0.271776 1.856 0.063
L9.jp_unemp -402.315312 1681.527338 -0.239 0.811
L9.jp_gdp 0.013279 0.032267 0.412 0.681
L9.usdjpy -274.113842 113.668559 -2.412 0.016
L10.tokyo -2.470682 1.514005 -1.632 0.103
L10.oil 43.316689 25.193604 1.719 0.086
L10.n225 0.151665 0.272114 0.557 0.577
L10.jp_unemp 1155.594948 1682.880607 0.687 0.492
L10.jp_gdp -0.015464 0.031875 -0.485 0.628
L10.usdjpy -10.696199 114.149775 -0.094 0.925
L11.tokyo -1.556199 1.512620 -1.029 0.304
L11.oil 14.518398 25.146522 0.577 0.564
L11.n225 1.200059 0.270559 4.435 0.000
L11.jp_unemp 1541.942372 1682.668164 0.916 0.359
L11.jp_gdp -0.013332 0.031715 -0.420 0.674
L11.usdjpy -487.008962 114.294037 -4.261 0.000
L12.tokyo 1.520000 1.525886 0.996 0.319
L12.oil -13.679541 24.968858 -0.548 0.584
L12.n225 0.412218 0.275009 1.499 0.134
L12.jp_unemp -1.400201 1682.018334 -0.001 0.999
L12.jp_gdp -0.031785 0.031137 -1.021 0.307
L12.usdjpy -185.869922 114.965387 -1.617 0.106
L13.tokyo 2.292035 1.523669 1.504 0.133
L13.oil -25.431535 24.407836 -1.042 0.297
L13.n225 0.283730 0.271747 1.044 0.296
L13.jp_unemp 1018.470439 1681.062840 0.606 0.545
L13.jp_gdp 0.002688 0.031157 0.086 0.931
L13.usdjpy -69.938741 114.028106 -0.613 0.540
L14.tokyo -1.081139 1.528495 -0.707 0.479
L14.oil 7.790325 22.356626 0.348 0.727
L14.n225 0.191125 0.268278 0.712 0.476
L14.jp_unemp -414.968883 1687.255880 -0.246 0.806
L14.jp_gdp 0.002682 0.038379 0.070 0.944
L14.usdjpy -380.841542 111.629170 -3.412 0.001
===============================================================================
Results for equation usdjpy
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const -0.012941 0.016903 -0.766 0.444
L1.tokyo 0.000344 0.000482 0.715 0.475
L1.oil 0.017695 0.008922 1.983 0.047
L1.n225 0.000281 0.000084 3.335 0.001
L1.jp_unemp -0.115165 0.544672 -0.211 0.833
L1.jp_gdp 0.000003 0.000011 0.277 0.782
L1.usdjpy -0.085370 0.036342 -2.349 0.019
L2.tokyo -0.000238 0.000482 -0.493 0.622
L2.oil 0.000051 0.008305 0.006 0.995
L2.n225 -0.000140 0.000085 -1.645 0.100
L2.jp_unemp 0.125219 0.544516 0.230 0.818
L2.jp_gdp 0.000003 0.000011 0.284 0.776
L2.usdjpy 0.068602 0.036845 1.862 0.063
L3.tokyo 0.000657 0.000483 1.360 0.174
L3.oil 0.002288 0.008298 0.276 0.783
L3.n225 -0.000189 0.000086 -2.205 0.027
L3.jp_unemp 1.020634 0.544596 1.874 0.061
L3.jp_gdp -0.000008 0.000011 -0.691 0.490
L3.usdjpy 0.028265 0.036858 0.767 0.443
L4.tokyo 0.000414 0.000488 0.848 0.397
L4.oil 0.006152 0.008191 0.751 0.453
L4.n225 -0.000087 0.000086 -1.009 0.313
L4.jp_unemp 1.120773 0.544392 2.059 0.040
L4.jp_gdp -0.000000 0.000011 -0.046 0.963
L4.usdjpy -0.010167 0.036898 -0.276 0.783
L5.tokyo 0.000161 0.000489 0.329 0.742
L5.oil -0.000214 0.008190 -0.026 0.979
L5.n225 -0.000025 0.000087 -0.293 0.769
L5.jp_unemp -1.090493 0.545222 -2.000 0.045
L5.jp_gdp 0.000019 0.000011 1.724 0.085
L5.usdjpy -0.087724 0.036711 -2.390 0.017
L6.tokyo -0.000150 0.000490 -0.306 0.760
L6.oil 0.005532 0.008135 0.680 0.496
L6.n225 -0.000014 0.000086 -0.168 0.867
L6.jp_unemp -0.361075 0.545351 -0.662 0.508
L6.jp_gdp -0.000002 0.000011 -0.183 0.855
L6.usdjpy 0.040665 0.036617 1.111 0.267
L7.tokyo 0.000452 0.000492 0.919 0.358
L7.oil 0.013116 0.008119 1.616 0.106
L7.n225 0.000199 0.000088 2.273 0.023
L7.jp_unemp 0.414658 0.543997 0.762 0.446
L7.jp_gdp -0.000007 0.000011 -0.693 0.488
L7.usdjpy -0.093690 0.036308 -2.580 0.010
L8.tokyo -0.000312 0.000490 -0.636 0.525
L8.oil -0.005389 0.008170 -0.660 0.510
L8.n225 0.000094 0.000087 1.074 0.283
L8.jp_unemp 0.628284 0.543631 1.156 0.248
L8.jp_gdp 0.000000 0.000011 0.034 0.973
L8.usdjpy -0.119717 0.036397 -3.289 0.001
L9.tokyo 0.000007 0.000491 0.014 0.989
L9.oil 0.003111 0.008196 0.380 0.704
L9.n225 0.000024 0.000088 0.274 0.784
L9.jp_unemp -0.840016 0.543880 -1.544 0.122
L9.jp_gdp 0.000002 0.000010 0.183 0.855
L9.usdjpy -0.025266 0.036765 -0.687 0.492
L10.tokyo 0.000047 0.000490 0.095 0.924
L10.oil -0.002782 0.008149 -0.341 0.733
L10.n225 0.000066 0.000088 0.754 0.451
L10.jp_unemp -0.193857 0.544318 -0.356 0.722
L10.jp_gdp 0.000009 0.000010 0.826 0.409
L10.usdjpy -0.085370 0.036921 -2.312 0.021
L11.tokyo 0.000928 0.000489 1.898 0.058
L11.oil -0.005138 0.008133 -0.632 0.528
L11.n225 -0.000148 0.000088 -1.690 0.091
L11.jp_unemp 0.275886 0.544249 0.507 0.612
L11.jp_gdp -0.000016 0.000010 -1.588 0.112
L11.usdjpy -0.008751 0.036968 -0.237 0.813
L12.tokyo -0.000358 0.000494 -0.724 0.469
L12.oil -0.003223 0.008076 -0.399 0.690
L12.n225 -0.000079 0.000089 -0.888 0.374
L12.jp_unemp 0.024623 0.544039 0.045 0.964
L12.jp_gdp -0.000004 0.000010 -0.428 0.669
L12.usdjpy 0.027025 0.037185 0.727 0.467
L13.tokyo 0.000744 0.000493 1.509 0.131
L13.oil 0.003377 0.007895 0.428 0.669
L13.n225 -0.000037 0.000088 -0.418 0.676
L13.jp_unemp 0.243576 0.543730 0.448 0.654
L13.jp_gdp 0.000010 0.000010 1.035 0.300
L13.usdjpy 0.039907 0.036882 1.082 0.279
L14.tokyo -0.000213 0.000494 -0.432 0.666
L14.oil 0.015281 0.007231 2.113 0.035
L14.n225 -0.000013 0.000087 -0.152 0.879
L14.jp_unemp 0.638267 0.545733 1.170 0.242
L14.jp_gdp -0.000016 0.000012 -1.322 0.186
L14.usdjpy 0.003866 0.036106 0.107 0.915
===============================================================================
Correlation matrix of residuals
tokyo oil n225 jp_unemp jp_gdp usdjpy
tokyo 1.000000 0.064813 0.401044 -0.095193 0.020429 0.108723
oil 0.064813 1.000000 0.164610 -0.004623 -0.006896 0.054685
n225 0.401044 0.164610 1.000000 -0.088557 0.062500 0.372348
jp_unemp -0.095193 -0.004623 -0.088557 1.000000 0.076504 -0.017203
jp_gdp 0.020429 -0.006896 0.062500 0.076504 1.000000 -0.023702
usdjpy 0.108723 0.054685 0.372348 -0.017203 -0.023702 1.000000
# Calculate yhat
yhat_var2_5 = var2_model5.forecast(var_model5.endog, steps=len(test_diff5))
# clean up yhat
df_yhat_var2_5 = pd.DataFrame(yhat_var2_5)
df_yhat_var2_5.index = test_diff5.index
df_yhat_var2_5.columns = test_diff5.columns
df_yhat_var2_5
| tokyo | oil | n225 | jp_unemp | jp_gdp | usdjpy | |
|---|---|---|---|---|---|---|
| Date | ||||||
| 2021-01-04 | 2.372628 | 0.287518 | 73.167477 | 0.002115 | -482.781396 | -0.175379 |
| 2021-01-05 | 0.345862 | 0.294738 | 11.520023 | 0.004589 | -31.465788 | -0.128185 |
| 2021-01-06 | 5.309567 | 0.067789 | 25.945321 | -0.005274 | 22.596011 | 0.019328 |
| 2021-01-07 | -5.928458 | 0.196840 | -2.026457 | 0.000696 | -961.812042 | -0.116844 |
| 2021-01-08 | 3.009113 | 0.635541 | 18.285359 | -0.000467 | -802.567411 | 0.106791 |
| ... | ... | ... | ... | ... | ... | ... |
| 2021-11-29 | -0.285417 | -0.002314 | 8.825160 | 0.000091 | -2.882649 | -0.010832 |
| 2021-11-30 | -0.285417 | -0.002314 | 8.825163 | 0.000091 | -2.882663 | -0.010832 |
| 2021-12-01 | -0.285417 | -0.002314 | 8.825166 | 0.000091 | -2.882650 | -0.010832 |
| 2021-12-02 | -0.285417 | -0.002314 | 8.825165 | 0.000091 | -2.882632 | -0.010832 |
| 2021-12-03 | -0.285417 | -0.002314 | 8.825165 | 0.000091 | -2.882650 | -0.010832 |
226 rows × 6 columns
# Recover first order differencing
df_pred_var2_5 = pd.concat([tokyo_5[tokyo_5.index==pd.datetime(2020,12,30)], df_yhat_var2_5], axis=0).cumsum()
df_pred_var2_5 = df_pred_var2_5[df_pred_var2_5.index!=pd.datetime(2020,12,30)]
y_pred_var2_5 = df_pred_var2_5["tokyo"]
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_var2_5.plot(ax=ax, label="VAR(14) with Oil, N225, Unemployement, GDP, USD-JPY", linestyle="dashed")
plt.legend()
plt.show()
var2_metrics5 = calc_metrics(y_test, y_pred_var2_5, return_dict=True)
Test Error Metrics MAE: 149.9029 MSE: 33680.3447 MAPE: 0.072 sMAPE: 0.0684 MASE: -6.441
# Merge data and fill NAs with interpolation
tokyo_6 = pd.concat([tokyo, gdp_s, unemp_s, oil_s, usdjpy_s, interest_s, n225_s], axis=1)
tokyo_6["jp_gdp"] = tokyo_6["jp_gdp"].fillna(method="ffill")
tokyo_6["jp_unemp"] = tokyo_6["jp_unemp"].fillna(method="ffill")
tokyo_6 = tokyo_6.dropna(subset=["tokyo"])
tokyo_6["oil"] = tokyo_6["oil"].interpolate()
tokyo_6["n225"] = tokyo_6["n225"].interpolate()
tokyo_6["usdjpy"] = tokyo_6["usdjpy"].interpolate()
tokyo_6[tokyo_6.isnull().any(axis=1)]
| tokyo | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| Date |
# Take first difference
tokyo_6_diff = tokyo_6.diff().dropna()
# Split diff into train/test
train_diff6 = tokyo_6_diff[tokyo_6_diff.index.year!=2021]
test_diff6 = tokyo_6_diff[tokyo_6_diff.index.year==2021]
from statsmodels.tsa.vector_ar.var_model import VAR
var_model6=VAR(train_diff)
results_aic6 =[]
for p in range(1,21):
results6 = var_model6.fit(p)
results_aic6.append(results6.aic)
# Min=2
plt.plot(list(np.arange(1,21,1)), results_aic, marker="o")
plt.xlabel("Order")
plt.ylabel("AIC")
plt.show()
var2_model6 = var_model6.fit(15)
print(var2_model6.summary())
Summary of Regression Results
==================================
Model: VAR
Method: OLS
Date: Wed, 01, Jun, 2022
Time: 18:49:05
--------------------------------------------------------------------
No. of Equations: 7.00000 BIC: 20.9383
Nobs: 959.000 HQIC: 18.6073
Log likelihood: -17018.0 FPE: 2.89134e+07
AIC: 17.1735 Det(Omega_mle): 1.38798e+07
--------------------------------------------------------------------
Results for equation tokyo
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 0.765500 1.287345 0.595 0.552
L1.tokyo 0.018599 0.037006 0.503 0.615
L1.jp_gdp 0.000939 0.000856 1.097 0.273
L1.jp_unemp -24.482515 41.668023 -0.588 0.557
L1.oil 0.065695 0.722961 0.091 0.928
L1.usdjpy 4.839987 2.938789 1.647 0.100
L1.jp_i -70.211361 113.014763 -0.621 0.534
L1.n225 -0.004690 0.006671 -0.703 0.482
L2.tokyo -0.059075 0.036956 -1.599 0.110
L2.jp_gdp -0.000248 0.000855 -0.290 0.772
L2.jp_unemp 29.081276 41.526120 0.700 0.484
L2.oil -0.111585 0.714270 -0.156 0.876
L2.usdjpy -0.175828 2.992989 -0.059 0.953
L2.jp_i -43.058902 113.262907 -0.380 0.704
L2.n225 0.002570 0.006756 0.380 0.704
L3.tokyo -0.103305 0.036983 -2.793 0.005
L3.jp_gdp -0.000446 0.000850 -0.525 0.599
L3.jp_unemp 21.894382 41.528052 0.527 0.598
L3.oil -0.088623 0.656726 -0.135 0.893
L3.usdjpy 0.260057 2.988348 0.087 0.931
L3.jp_i -40.087496 114.024124 -0.352 0.725
L3.n225 0.005448 0.006770 0.805 0.421
L4.tokyo -0.092221 0.037522 -2.458 0.014
L4.jp_gdp 0.000230 0.000844 0.272 0.785
L4.jp_unemp -19.134873 41.623662 -0.460 0.646
L4.oil 1.182748 0.642743 1.840 0.066
L4.usdjpy 2.805730 2.972899 0.944 0.345
L4.jp_i -19.501817 114.494178 -0.170 0.865
L4.n225 -0.003245 0.006775 -0.479 0.632
L5.tokyo 0.082068 0.037493 2.189 0.029
L5.jp_gdp -0.000792 0.000833 -0.951 0.342
L5.jp_unemp 16.807841 41.614882 0.404 0.686
L5.oil 1.620691 0.631048 2.568 0.010
L5.usdjpy -5.155253 2.989190 -1.725 0.085
L5.jp_i 155.005676 114.754355 1.351 0.177
L5.n225 -0.001848 0.006826 -0.271 0.787
L6.tokyo -0.082641 0.037714 -2.191 0.028
L6.jp_gdp 0.000385 0.000840 0.459 0.647
L6.jp_unemp -38.744694 41.717174 -0.929 0.353
L6.oil 0.341885 0.628071 0.544 0.586
L6.usdjpy 1.174448 2.987053 0.393 0.694
L6.jp_i 179.410995 114.847149 1.562 0.118
L6.n225 0.007024 0.006883 1.021 0.307
L7.tokyo 0.015334 0.037775 0.406 0.685
L7.jp_gdp -0.000906 0.000832 -1.089 0.276
L7.jp_unemp -11.817876 41.645759 -0.284 0.777
L7.oil 0.807383 0.623749 1.294 0.196
L7.usdjpy -2.195959 2.960645 -0.742 0.458
L7.jp_i 185.905786 114.984291 1.617 0.106
L7.n225 -0.009272 0.006928 -1.338 0.181
L8.tokyo -0.040240 0.037696 -1.067 0.286
L8.jp_gdp 0.000885 0.000826 1.070 0.284
L8.jp_unemp -0.696048 41.510859 -0.017 0.987
L8.oil -0.339195 0.627801 -0.540 0.589
L8.usdjpy 1.792026 2.969455 0.603 0.546
L8.jp_i -136.665851 115.724103 -1.181 0.238
L8.n225 -0.014145 0.006958 -2.033 0.042
L9.tokyo -0.010814 0.037576 -0.288 0.774
L9.jp_gdp -0.001627 0.000816 -1.994 0.046
L9.jp_unemp -2.491492 41.485082 -0.060 0.952
L9.oil 0.523349 0.630203 0.830 0.406
L9.usdjpy 9.357711 2.962546 3.159 0.002
L9.jp_i 146.199856 115.445505 1.266 0.205
L9.n225 -0.021935 0.006912 -3.174 0.002
L10.tokyo 0.006590 0.037513 0.176 0.861
L10.jp_gdp -0.000295 0.000807 -0.366 0.715
L10.jp_unemp -8.376963 41.634355 -0.201 0.841
L10.oil 0.745498 0.629982 1.183 0.237
L10.usdjpy 1.267971 2.985562 0.425 0.671
L10.jp_i 137.386783 116.430503 1.180 0.238
L10.n225 -0.013811 0.006958 -1.985 0.047
L11.tokyo 0.065588 0.037379 1.755 0.079
L11.jp_gdp 0.000230 0.000790 0.292 0.770
L11.jp_unemp 23.105471 41.622392 0.555 0.579
L11.oil 0.192635 0.624680 0.308 0.758
L11.usdjpy 4.067862 2.967889 1.371 0.170
L11.jp_i -209.286914 116.429171 -1.798 0.072
L11.n225 -0.009614 0.006950 -1.383 0.167
L12.tokyo -0.046494 0.037813 -1.230 0.219
L12.jp_gdp 0.000048 0.000787 0.061 0.951
L12.jp_unemp -8.794786 41.589055 -0.211 0.833
L12.oil -0.504642 0.623642 -0.809 0.418
L12.usdjpy -1.471068 2.982102 -0.493 0.622
L12.jp_i 36.541803 116.671376 0.313 0.754
L12.n225 0.005704 0.007011 0.813 0.416
L13.tokyo -0.002070 0.037681 -0.055 0.956
L13.jp_gdp -0.000668 0.000775 -0.862 0.389
L13.jp_unemp -2.373335 41.589082 -0.057 0.954
L13.oil -0.457385 0.618246 -0.740 0.459
L13.usdjpy -4.003487 2.948699 -1.358 0.175
L13.jp_i 178.815053 117.169023 1.526 0.127
L13.n225 0.002375 0.007003 0.339 0.735
L14.tokyo 0.032230 0.037720 0.854 0.393
L14.jp_gdp -0.000950 0.000971 -0.979 0.328
L14.jp_unemp 6.110224 41.943320 0.146 0.884
L14.oil -0.030356 0.604672 -0.050 0.960
L14.usdjpy 1.045376 2.920790 0.358 0.720
L14.jp_i 165.893685 116.998115 1.418 0.156
L14.n225 -0.015803 0.006954 -2.272 0.023
L15.tokyo 0.103335 0.037868 2.729 0.006
L15.jp_gdp 0.000587 0.001008 0.582 0.560
L15.jp_unemp -5.840750 42.030339 -0.139 0.889
L15.oil -0.396975 0.556007 -0.714 0.475
L15.usdjpy 7.286317 2.854421 2.553 0.011
L15.jp_i -126.233884 116.325443 -1.085 0.278
L15.n225 -0.024442 0.006819 -3.584 0.000
===============================================================================
Results for equation jp_gdp
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const -17.007450 50.783398 -0.335 0.738
L1.tokyo 2.898491 1.459833 1.985 0.047
L1.jp_gdp -0.043520 0.033766 -1.289 0.197
L1.jp_unemp -2.772204 1643.726705 -0.002 0.999
L1.oil -28.491569 28.519497 -0.999 0.318
L1.usdjpy -135.592659 115.929810 -1.170 0.242
L1.jp_i -2163.963979 4458.224076 -0.485 0.627
L1.n225 -0.041151 0.263159 -0.156 0.876
L2.tokyo 0.279921 1.457854 0.192 0.848
L2.jp_gdp -0.036397 0.033734 -1.079 0.281
L2.jp_unemp -421.223476 1638.128882 -0.257 0.797
L2.oil 44.172094 28.176654 1.568 0.117
L2.usdjpy 280.371908 118.067903 2.375 0.018
L2.jp_i -2049.545811 4468.012880 -0.459 0.646
L2.n225 -0.172870 0.266506 -0.649 0.517
L3.tokyo -4.415699 1.458919 -3.027 0.002
L3.jp_gdp -0.039572 0.033533 -1.180 0.238
L3.jp_unemp -143.014812 1638.205091 -0.087 0.930
L3.oil 55.292149 25.906636 2.134 0.033
L3.usdjpy 218.736427 117.884830 1.856 0.064
L3.jp_i -3004.766925 4498.041500 -0.668 0.504
L3.n225 0.013487 0.267072 0.050 0.960
L4.tokyo 1.295774 1.480186 0.875 0.381
L4.jp_gdp 0.016084 0.033305 0.483 0.629
L4.jp_unemp -111.184664 1641.976750 -0.068 0.946
L4.oil 39.986709 25.355012 1.577 0.115
L4.usdjpy 15.922992 117.275379 0.136 0.892
L4.jp_i -1719.765713 4516.584252 -0.381 0.703
L4.n225 0.471891 0.267264 1.766 0.077
L5.tokyo -3.499918 1.479029 -2.366 0.018
L5.jp_gdp 0.022106 0.032868 0.673 0.501
L5.jp_unemp -1914.865460 1641.630385 -1.166 0.243
L5.oil 10.189144 24.893661 0.409 0.682
L5.usdjpy 87.549468 117.918048 0.742 0.458
L5.jp_i 6069.022236 4526.847757 1.341 0.180
L5.n225 -0.973228 0.269276 -3.614 0.000
L6.tokyo 5.424702 1.487737 3.646 0.000
L6.jp_gdp 0.043535 0.033126 1.314 0.189
L6.jp_unemp -1270.640563 1645.665601 -0.772 0.440
L6.oil 19.436848 24.776225 0.784 0.433
L6.usdjpy 176.673591 117.833737 1.499 0.134
L6.jp_i 3706.665373 4530.508326 0.818 0.413
L6.n225 -1.282008 0.271504 -4.722 0.000
L7.tokyo 0.275020 1.490158 0.185 0.854
L7.jp_gdp -0.003007 0.032810 -0.092 0.927
L7.jp_unemp -253.978324 1642.848407 -0.155 0.877
L7.oil 46.879887 24.605765 1.905 0.057
L7.usdjpy -332.993600 116.791969 -2.851 0.004
L7.jp_i 9460.238010 4535.918311 2.086 0.037
L7.n225 -0.411634 0.273287 -1.506 0.132
L8.tokyo -2.687254 1.487023 -1.807 0.071
L8.jp_gdp 0.056054 0.032599 1.720 0.086
L8.jp_unemp -129.751054 1637.526864 -0.079 0.937
L8.oil 19.895238 24.765579 0.803 0.422
L8.usdjpy -86.086628 117.139513 -0.735 0.462
L8.jp_i -9204.190388 4565.102544 -2.016 0.044
L8.n225 0.331474 0.274483 1.208 0.227
L9.tokyo -1.676090 1.482291 -1.131 0.258
L9.jp_gdp 0.016558 0.032191 0.514 0.607
L9.jp_unemp -451.092892 1636.510024 -0.276 0.783
L9.oil 39.272362 24.860329 1.580 0.114
L9.usdjpy -106.198695 116.866964 -0.909 0.364
L9.jp_i -11641.364784 4554.112356 -2.556 0.011
L9.n225 0.450838 0.272658 1.653 0.098
L10.tokyo -2.299631 1.479823 -1.554 0.120
L10.jp_gdp 0.005854 0.031818 0.184 0.854
L10.jp_unemp 678.050044 1642.398564 0.413 0.680
L10.oil 34.361509 24.851625 1.383 0.167
L10.usdjpy -3.945763 117.774898 -0.034 0.973
L10.jp_i -6979.735617 4592.968713 -1.520 0.129
L10.n225 0.020164 0.274462 0.073 0.941
L11.tokyo -1.687552 1.474527 -1.144 0.252
L11.jp_gdp -0.004969 0.031159 -0.159 0.873
L11.jp_unemp 1606.196714 1641.926656 0.978 0.328
L11.oil 9.354562 24.642470 0.380 0.704
L11.usdjpy -307.727526 117.077769 -2.628 0.009
L11.jp_i -3742.856839 4592.916162 -0.815 0.415
L11.n225 1.006287 0.274174 3.670 0.000
L12.tokyo 1.011879 1.491654 0.678 0.498
L12.jp_gdp -0.025858 0.031065 -0.832 0.405
L12.jp_unemp 162.434594 1640.611570 0.099 0.921
L12.oil -20.629511 24.601511 -0.839 0.402
L12.usdjpy -127.053358 117.638437 -1.080 0.280
L12.jp_i -16061.973197 4602.470717 -3.490 0.000
L12.n225 0.603207 0.276590 2.181 0.029
L13.tokyo 2.159932 1.486462 1.453 0.146
L13.jp_gdp -0.002076 0.030575 -0.068 0.946
L13.jp_unemp 703.950795 1640.612604 0.429 0.668
L13.oil -36.256252 24.388675 -1.487 0.137
L13.usdjpy -0.220838 116.320752 -0.002 0.998
L13.jp_i -12865.836695 4622.101960 -2.784 0.005
L13.n225 0.397846 0.276252 1.440 0.150
L14.tokyo -1.129426 1.487989 -0.759 0.448
L14.jp_gdp 0.008589 0.038287 0.224 0.822
L14.jp_unemp -594.767147 1654.586668 -0.359 0.719
L14.oil 6.914532 23.853205 0.290 0.772
L14.usdjpy -339.374926 115.219764 -2.945 0.003
L14.jp_i 4571.987608 4615.359969 0.991 0.322
L14.n225 -0.026862 0.274340 -0.098 0.922
L15.tokyo 0.647118 1.493820 0.433 0.665
L15.jp_gdp -0.063748 0.039780 -1.602 0.109
L15.jp_unemp 1227.556865 1658.019393 0.740 0.459
L15.oil -5.058086 21.933441 -0.231 0.818
L15.usdjpy 87.754537 112.601649 0.779 0.436
L15.jp_i -27466.609838 4588.824291 -5.986 0.000
L15.n225 0.282961 0.269004 1.052 0.293
===============================================================================
Results for equation jp_unemp
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 0.000400 0.001066 0.375 0.708
L1.tokyo 0.000005 0.000031 0.173 0.863
L1.jp_gdp 0.000001 0.000001 0.786 0.432
L1.jp_unemp -0.001695 0.034515 -0.049 0.961
L1.oil 0.000703 0.000599 1.173 0.241
L1.usdjpy 0.005996 0.002434 2.463 0.014
L1.jp_i 0.075996 0.093614 0.812 0.417
L1.n225 -0.000005 0.000006 -0.995 0.320
L2.tokyo 0.000006 0.000031 0.187 0.852
L2.jp_gdp 0.000000 0.000001 0.306 0.760
L2.jp_unemp -0.004652 0.034397 -0.135 0.892
L2.oil 0.000272 0.000592 0.460 0.645
L2.usdjpy -0.000070 0.002479 -0.028 0.978
L2.jp_i -0.003421 0.093819 -0.036 0.971
L2.n225 -0.000007 0.000006 -1.180 0.238
L3.tokyo 0.000004 0.000031 0.129 0.898
L3.jp_gdp 0.000000 0.000001 0.295 0.768
L3.jp_unemp 0.000224 0.034399 0.007 0.995
L3.oil 0.000215 0.000544 0.395 0.693
L3.usdjpy 0.001047 0.002475 0.423 0.672
L3.jp_i -0.128947 0.094450 -1.365 0.172
L3.n225 0.000007 0.000006 1.170 0.242
L4.tokyo 0.000040 0.000031 1.278 0.201
L4.jp_gdp -0.000000 0.000001 -0.123 0.902
L4.jp_unemp -0.004987 0.034478 -0.145 0.885
L4.oil -0.000186 0.000532 -0.349 0.727
L4.usdjpy 0.002990 0.002463 1.214 0.225
L4.jp_i -0.091702 0.094839 -0.967 0.334
L4.n225 -0.000008 0.000006 -1.409 0.159
L5.tokyo -0.000010 0.000031 -0.311 0.756
L5.jp_gdp -0.000000 0.000001 -0.027 0.979
L5.jp_unemp -0.004126 0.034471 -0.120 0.905
L5.oil 0.000048 0.000523 0.093 0.926
L5.usdjpy -0.001279 0.002476 -0.517 0.605
L5.jp_i -0.089151 0.095055 -0.938 0.348
L5.n225 -0.000002 0.000006 -0.425 0.671
L6.tokyo 0.000017 0.000031 0.559 0.576
L6.jp_gdp 0.000000 0.000001 0.544 0.587
L6.jp_unemp 0.004797 0.034556 0.139 0.890
L6.oil -0.000048 0.000520 -0.093 0.926
L6.usdjpy -0.001728 0.002474 -0.699 0.485
L6.jp_i -0.049869 0.095131 -0.524 0.600
L6.n225 0.000005 0.000006 0.872 0.383
L7.tokyo -0.000041 0.000031 -1.300 0.193
L7.jp_gdp -0.000000 0.000001 -0.510 0.610
L7.jp_unemp -0.002614 0.034496 -0.076 0.940
L7.oil 0.000065 0.000517 0.126 0.900
L7.usdjpy 0.001699 0.002452 0.693 0.488
L7.jp_i -0.065119 0.095245 -0.684 0.494
L7.n225 0.000001 0.000006 0.238 0.812
L8.tokyo -0.000011 0.000031 -0.357 0.721
L8.jp_gdp -0.000000 0.000001 -0.511 0.610
L8.jp_unemp -0.007219 0.034385 -0.210 0.834
L8.oil -0.001284 0.000520 -2.469 0.014
L8.usdjpy 0.001389 0.002460 0.565 0.572
L8.jp_i -0.081417 0.095858 -0.849 0.396
L8.n225 -0.000003 0.000006 -0.550 0.583
L9.tokyo 0.000047 0.000031 1.505 0.132
L9.jp_gdp -0.000000 0.000001 -0.442 0.658
L9.jp_unemp 0.004147 0.034363 0.121 0.904
L9.oil -0.000635 0.000522 -1.217 0.224
L9.usdjpy 0.003732 0.002454 1.521 0.128
L9.jp_i -0.082910 0.095627 -0.867 0.386
L9.n225 -0.000005 0.000006 -0.906 0.365
L10.tokyo 0.000019 0.000031 0.613 0.540
L10.jp_gdp 0.000000 0.000001 0.220 0.826
L10.jp_unemp -0.000994 0.034487 -0.029 0.977
L10.oil -0.000270 0.000522 -0.518 0.604
L10.usdjpy 0.000641 0.002473 0.259 0.795
L10.jp_i -0.030627 0.096443 -0.318 0.751
L10.n225 -0.000006 0.000006 -1.041 0.298
L11.tokyo 0.000009 0.000031 0.306 0.759
L11.jp_gdp -0.000000 0.000001 -0.159 0.874
L11.jp_unemp -0.001897 0.034477 -0.055 0.956
L11.oil -0.000273 0.000517 -0.527 0.598
L11.usdjpy 0.003200 0.002458 1.302 0.193
L11.jp_i -0.113660 0.096442 -1.179 0.239
L11.n225 0.000004 0.000006 0.618 0.537
L12.tokyo -0.000023 0.000031 -0.721 0.471
L12.jp_gdp 0.000000 0.000001 0.254 0.799
L12.jp_unemp -0.002596 0.034449 -0.075 0.940
L12.oil 0.000033 0.000517 0.063 0.949
L12.usdjpy 0.000337 0.002470 0.136 0.891
L12.jp_i -0.040731 0.096642 -0.421 0.673
L12.n225 0.000001 0.000006 0.200 0.841
L13.tokyo 0.000013 0.000031 0.425 0.671
L13.jp_gdp 0.000000 0.000001 0.039 0.968
L13.jp_unemp -0.003525 0.034449 -0.102 0.919
L13.oil -0.000235 0.000512 -0.458 0.647
L13.usdjpy -0.001187 0.002442 -0.486 0.627
L13.jp_i -0.047678 0.097055 -0.491 0.623
L13.n225 -0.000000 0.000006 -0.058 0.954
L14.tokyo -0.000023 0.000031 -0.739 0.460
L14.jp_gdp -0.000001 0.000001 -0.920 0.357
L14.jp_unemp -0.007685 0.034743 -0.221 0.825
L14.oil -0.000090 0.000501 -0.180 0.857
L14.usdjpy 0.001076 0.002419 0.445 0.657
L14.jp_i 0.086062 0.096913 0.888 0.375
L14.n225 -0.000009 0.000006 -1.501 0.133
L15.tokyo -0.000027 0.000031 -0.873 0.382
L15.jp_gdp 0.000001 0.000001 0.704 0.481
L15.jp_unemp -0.003340 0.034815 -0.096 0.924
L15.oil -0.000197 0.000461 -0.427 0.669
L15.usdjpy -0.003916 0.002364 -1.656 0.098
L15.jp_i 0.033500 0.096356 0.348 0.728
L15.n225 0.000008 0.000006 1.490 0.136
===============================================================================
Results for equation oil
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const -0.062749 0.061285 -1.024 0.306
L1.tokyo -0.001164 0.001762 -0.661 0.509
L1.jp_gdp -0.000043 0.000041 -1.059 0.290
L1.jp_unemp 0.322426 1.983645 0.163 0.871
L1.oil -0.289795 0.034417 -8.420 0.000
L1.usdjpy -0.102528 0.139904 -0.733 0.464
L1.jp_i 1.369191 5.380174 0.254 0.799
L1.n225 -0.000023 0.000318 -0.071 0.943
L2.tokyo 0.001740 0.001759 0.989 0.323
L2.jp_gdp -0.000049 0.000041 -1.193 0.233
L2.jp_unemp 1.130870 1.976890 0.572 0.567
L2.oil -0.098452 0.034004 -2.895 0.004
L2.usdjpy -0.127650 0.142484 -0.896 0.370
L2.jp_i -0.943841 5.391987 -0.175 0.861
L2.n225 0.000680 0.000322 2.113 0.035
L3.tokyo -0.002867 0.001761 -1.628 0.103
L3.jp_gdp -0.000004 0.000040 -0.106 0.915
L3.jp_unemp -0.322470 1.976982 -0.163 0.870
L3.oil -0.067987 0.031264 -2.175 0.030
L3.usdjpy -0.045487 0.142263 -0.320 0.749
L3.jp_i -2.910279 5.428226 -0.536 0.592
L3.n225 0.000346 0.000322 1.073 0.283
L4.tokyo -0.000727 0.001786 -0.407 0.684
L4.jp_gdp 0.000048 0.000040 1.185 0.236
L4.jp_unemp -0.397350 1.981534 -0.201 0.841
L4.oil -0.055541 0.030598 -1.815 0.069
L4.usdjpy -0.190623 0.141528 -1.347 0.178
L4.jp_i -3.945133 5.450603 -0.724 0.469
L4.n225 0.000656 0.000323 2.034 0.042
L5.tokyo -0.002472 0.001785 -1.385 0.166
L5.jp_gdp 0.000018 0.000040 0.455 0.649
L5.jp_unemp -0.849547 1.981116 -0.429 0.668
L5.oil 0.018666 0.030042 0.621 0.534
L5.usdjpy 0.014896 0.142303 0.105 0.917
L5.jp_i -5.002603 5.462989 -0.916 0.360
L5.n225 0.000675 0.000325 2.077 0.038
L6.tokyo -0.002473 0.001795 -1.378 0.168
L6.jp_gdp 0.000017 0.000040 0.423 0.672
L6.jp_unemp -0.495242 1.985985 -0.249 0.803
L6.oil 0.030863 0.029900 1.032 0.302
L6.usdjpy -0.110966 0.142201 -0.780 0.435
L6.jp_i 3.541255 5.467406 0.648 0.517
L6.n225 0.000595 0.000328 1.816 0.069
L7.tokyo 0.001192 0.001798 0.663 0.507
L7.jp_gdp 0.000060 0.000040 1.505 0.132
L7.jp_unemp -0.367868 1.982585 -0.186 0.853
L7.oil -0.006219 0.029694 -0.209 0.834
L7.usdjpy -0.186024 0.140944 -1.320 0.187
L7.jp_i 0.623799 5.473935 0.114 0.909
L7.n225 0.000558 0.000330 1.693 0.090
L8.tokyo -0.001108 0.001795 -0.617 0.537
L8.jp_gdp 0.000012 0.000039 0.311 0.756
L8.jp_unemp 0.303645 1.976163 0.154 0.878
L8.oil -0.014149 0.029887 -0.473 0.636
L8.usdjpy 0.013955 0.141364 0.099 0.921
L8.jp_i 5.145874 5.509155 0.934 0.350
L8.n225 0.000431 0.000331 1.301 0.193
L9.tokyo 0.001101 0.001789 0.615 0.538
L9.jp_gdp 0.000047 0.000039 1.217 0.224
L9.jp_unemp 0.695979 1.974936 0.352 0.725
L9.oil -0.025404 0.030001 -0.847 0.397
L9.usdjpy 0.120365 0.141035 0.853 0.393
L9.jp_i -8.103896 5.495892 -1.475 0.140
L9.n225 0.000126 0.000329 0.384 0.701
L10.tokyo 0.002059 0.001786 1.153 0.249
L10.jp_gdp 0.000023 0.000038 0.603 0.546
L10.jp_unemp 1.661874 1.982043 0.838 0.402
L10.oil -0.047962 0.029991 -1.599 0.110
L10.usdjpy -0.071662 0.142130 -0.504 0.614
L10.jp_i 1.447813 5.542784 0.261 0.794
L10.n225 -0.000085 0.000331 -0.258 0.796
L11.tokyo -0.005482 0.001779 -3.081 0.002
L11.jp_gdp 0.000012 0.000038 0.322 0.747
L11.jp_unemp 0.583721 1.981473 0.295 0.768
L11.oil -0.053273 0.029738 -1.791 0.073
L11.usdjpy -0.326701 0.141289 -2.312 0.021
L11.jp_i 8.561546 5.542720 1.545 0.122
L11.n225 0.000812 0.000331 2.454 0.014
L12.tokyo -0.000442 0.001800 -0.245 0.806
L12.jp_gdp 0.000005 0.000037 0.142 0.887
L12.jp_unemp -1.834297 1.979886 -0.926 0.354
L12.oil -0.052393 0.029689 -1.765 0.078
L12.usdjpy 0.010813 0.141966 0.076 0.939
L12.jp_i 2.997794 5.554251 0.540 0.589
L12.n225 0.000431 0.000334 1.291 0.197
L13.tokyo 0.000388 0.001794 0.216 0.829
L13.jp_gdp 0.000808 0.000037 21.889 0.000
L13.jp_unemp -7.751676 1.979887 -3.915 0.000
L13.oil 0.000853 0.029432 0.029 0.977
L13.usdjpy 0.010155 0.140376 0.072 0.942
L13.jp_i 2.574621 5.577941 0.462 0.644
L13.n225 0.000523 0.000333 1.568 0.117
L14.tokyo 0.000003 0.001796 0.002 0.999
L14.jp_gdp -0.000438 0.000046 -9.478 0.000
L14.jp_unemp 2.542084 1.996751 1.273 0.203
L14.oil -0.018400 0.028786 -0.639 0.523
L14.usdjpy 0.033597 0.139047 0.242 0.809
L14.jp_i -2.732869 5.569805 -0.491 0.624
L14.n225 -0.000226 0.000331 -0.683 0.495
L15.tokyo -0.001191 0.001803 -0.661 0.509
L15.jp_gdp -0.000189 0.000048 -3.946 0.000
L15.jp_unemp 1.252582 2.000894 0.626 0.531
L15.oil 0.004048 0.026469 0.153 0.878
L15.usdjpy 0.204737 0.135887 1.507 0.132
L15.jp_i -5.427839 5.537782 -0.980 0.327
L15.n225 -0.000087 0.000325 -0.269 0.788
===============================================================================
Results for equation usdjpy
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const -0.014737 0.016747 -0.880 0.379
L1.tokyo 0.000294 0.000481 0.611 0.541
L1.jp_gdp 0.000001 0.000011 0.081 0.935
L1.jp_unemp -0.089336 0.542054 -0.165 0.869
L1.oil 0.020585 0.009405 2.189 0.029
L1.usdjpy -0.096201 0.038230 -2.516 0.012
L1.jp_i 3.256948 1.470194 2.215 0.027
L1.n225 0.000245 0.000087 2.822 0.005
L2.tokyo -0.000480 0.000481 -0.998 0.318
L2.jp_gdp 0.000001 0.000011 0.070 0.944
L2.jp_unemp 0.124693 0.540208 0.231 0.817
L2.oil 0.000484 0.009292 0.052 0.958
L2.usdjpy 0.042147 0.038935 1.082 0.279
L2.jp_i 2.982425 1.473422 2.024 0.043
L2.n225 -0.000109 0.000088 -1.239 0.215
L3.tokyo 0.000578 0.000481 1.202 0.229
L3.jp_gdp -0.000011 0.000011 -1.029 0.304
L3.jp_unemp 1.136938 0.540233 2.105 0.035
L3.oil 0.002243 0.008543 0.263 0.793
L3.usdjpy 0.024188 0.038875 0.622 0.534
L3.jp_i -1.797688 1.483325 -1.212 0.226
L3.n225 -0.000095 0.000088 -1.076 0.282
L4.tokyo 0.000509 0.000488 1.044 0.297
L4.jp_gdp -0.000005 0.000011 -0.478 0.633
L4.jp_unemp 1.115966 0.541477 2.061 0.039
L4.oil 0.002759 0.008361 0.330 0.741
L4.usdjpy -0.036157 0.038674 -0.935 0.350
L4.jp_i 4.218768 1.489440 2.832 0.005
L4.n225 -0.000145 0.000088 -1.643 0.100
L5.tokyo 0.000108 0.000488 0.221 0.825
L5.jp_gdp 0.000015 0.000011 1.378 0.168
L5.jp_unemp -1.014484 0.541362 -1.874 0.061
L5.oil 0.001794 0.008209 0.219 0.827
L5.usdjpy -0.093479 0.038886 -2.404 0.016
L5.jp_i -1.154073 1.492824 -0.773 0.439
L5.n225 0.000018 0.000089 0.198 0.843
L6.tokyo -0.000043 0.000491 -0.087 0.930
L6.jp_gdp 0.000004 0.000011 0.355 0.723
L6.jp_unemp -0.209097 0.542693 -0.385 0.700
L6.oil 0.004831 0.008170 0.591 0.554
L6.usdjpy 0.021537 0.038858 0.554 0.579
L6.jp_i 3.939511 1.494031 2.637 0.008
L6.n225 -0.000037 0.000090 -0.409 0.683
L7.tokyo 0.000339 0.000491 0.690 0.490
L7.jp_gdp -0.000010 0.000011 -0.928 0.353
L7.jp_unemp 0.333681 0.541764 0.616 0.538
L7.oil 0.014012 0.008114 1.727 0.084
L7.usdjpy -0.102818 0.038515 -2.670 0.008
L7.jp_i 0.785871 1.495815 0.525 0.599
L7.n225 0.000190 0.000090 2.114 0.035
L8.tokyo -0.000325 0.000490 -0.662 0.508
L8.jp_gdp -0.000002 0.000011 -0.230 0.818
L8.jp_unemp 0.575247 0.540009 1.065 0.287
L8.oil -0.007469 0.008167 -0.915 0.360
L8.usdjpy -0.130041 0.038629 -3.366 0.001
L8.jp_i 0.380251 1.505439 0.253 0.801
L8.n225 0.000140 0.000091 1.547 0.122
L9.tokyo 0.000076 0.000489 0.155 0.877
L9.jp_gdp 0.000003 0.000011 0.326 0.745
L9.jp_unemp -0.922628 0.539674 -1.710 0.087
L9.oil 0.003116 0.008198 0.380 0.704
L9.usdjpy -0.040929 0.038539 -1.062 0.288
L9.jp_i 3.829285 1.501815 2.550 0.011
L9.n225 -0.000014 0.000090 -0.153 0.879
L10.tokyo -0.000034 0.000488 -0.069 0.945
L10.jp_gdp 0.000004 0.000010 0.398 0.691
L10.jp_unemp -0.326307 0.541616 -0.602 0.547
L10.oil -0.001013 0.008195 -0.124 0.902
L10.usdjpy -0.092409 0.038839 -2.379 0.017
L10.jp_i 1.878461 1.514629 1.240 0.215
L10.n225 0.000039 0.000091 0.426 0.670
L11.tokyo 0.000969 0.000486 1.993 0.046
L11.jp_gdp -0.000014 0.000010 -1.371 0.170
L11.jp_unemp 0.403228 0.541460 0.745 0.456
L11.oil -0.004754 0.008126 -0.585 0.559
L11.usdjpy 0.014005 0.038609 0.363 0.717
L11.jp_i -1.061309 1.514612 -0.701 0.483
L11.n225 -0.000113 0.000090 -1.255 0.210
L12.tokyo -0.000403 0.000492 -0.819 0.413
L12.jp_gdp -0.000004 0.000010 -0.372 0.710
L12.jp_unemp 0.104936 0.541026 0.194 0.846
L12.oil -0.005276 0.008113 -0.650 0.515
L12.usdjpy 0.033367 0.038794 0.860 0.390
L12.jp_i -1.060195 1.517762 -0.699 0.485
L12.n225 -0.000101 0.000091 -1.109 0.267
L13.tokyo 0.000717 0.000490 1.463 0.143
L13.jp_gdp 0.000010 0.000010 0.975 0.329
L13.jp_unemp 0.177135 0.541027 0.327 0.743
L13.oil 0.003628 0.008043 0.451 0.652
L13.usdjpy 0.056547 0.038359 1.474 0.140
L13.jp_i -2.621324 1.524236 -1.720 0.085
L13.n225 -0.000009 0.000091 -0.101 0.920
L14.tokyo -0.000109 0.000491 -0.222 0.825
L14.jp_gdp -0.000013 0.000013 -1.006 0.314
L14.jp_unemp 0.726663 0.545635 1.332 0.183
L14.oil 0.013209 0.007866 1.679 0.093
L14.usdjpy -0.004416 0.037996 -0.116 0.907
L14.jp_i 1.246133 1.522013 0.819 0.413
L14.n225 -0.000032 0.000090 -0.354 0.724
L15.tokyo -0.000704 0.000493 -1.428 0.153
L15.jp_gdp 0.000009 0.000013 0.721 0.471
L15.jp_unemp -0.351486 0.546767 -0.643 0.520
L15.oil -0.000080 0.007233 -0.011 0.991
L15.usdjpy -0.054338 0.037133 -1.463 0.143
L15.jp_i 0.140151 1.513262 0.093 0.926
L15.n225 0.000004 0.000089 0.042 0.967
===============================================================================
Results for equation jp_i
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const -0.000117 0.000417 -0.281 0.779
L1.tokyo 0.000020 0.000012 1.656 0.098
L1.jp_gdp 0.000001 0.000000 2.041 0.041
L1.jp_unemp -0.005470 0.013484 -0.406 0.685
L1.oil -0.000354 0.000234 -1.515 0.130
L1.usdjpy -0.000624 0.000951 -0.656 0.512
L1.jp_i -0.054469 0.036573 -1.489 0.136
L1.n225 -0.000005 0.000002 -2.490 0.013
L2.tokyo -0.000012 0.000012 -0.975 0.330
L2.jp_gdp 0.000000 0.000000 0.652 0.515
L2.jp_unemp -0.008007 0.013438 -0.596 0.551
L2.oil 0.000248 0.000231 1.075 0.282
L2.usdjpy 0.000757 0.000969 0.782 0.434
L2.jp_i -0.079413 0.036653 -2.167 0.030
L2.n225 -0.000002 0.000002 -0.822 0.411
L3.tokyo -0.000022 0.000012 -1.805 0.071
L3.jp_gdp 0.000000 0.000000 1.427 0.154
L3.jp_unemp 0.013189 0.013439 0.981 0.326
L3.oil 0.000023 0.000213 0.109 0.913
L3.usdjpy 0.000119 0.000967 0.123 0.902
L3.jp_i 0.060693 0.036899 1.645 0.100
L3.n225 0.000003 0.000002 1.529 0.126
L4.tokyo -0.000003 0.000012 -0.265 0.791
L4.jp_gdp -0.000000 0.000000 -0.132 0.895
L4.jp_unemp -0.005366 0.013470 -0.398 0.690
L4.oil -0.000019 0.000208 -0.091 0.927
L4.usdjpy -0.002184 0.000962 -2.271 0.023
L4.jp_i 0.000743 0.037052 0.020 0.984
L4.n225 0.000002 0.000002 0.939 0.348
L5.tokyo 0.000006 0.000012 0.493 0.622
L5.jp_gdp -0.000001 0.000000 -2.818 0.005
L5.jp_unemp -0.010724 0.013467 -0.796 0.426
L5.oil 0.000136 0.000204 0.668 0.504
L5.usdjpy -0.001639 0.000967 -1.694 0.090
L5.jp_i 0.010984 0.037136 0.296 0.767
L5.n225 0.000001 0.000002 0.671 0.502
L6.tokyo -0.000001 0.000012 -0.063 0.950
L6.jp_gdp -0.000000 0.000000 -0.453 0.650
L6.jp_unemp 0.006545 0.013500 0.485 0.628
L6.oil 0.000369 0.000203 1.816 0.069
L6.usdjpy -0.000059 0.000967 -0.061 0.951
L6.jp_i -0.038553 0.037166 -1.037 0.300
L6.n225 0.000003 0.000002 1.516 0.130
L7.tokyo 0.000003 0.000012 0.213 0.831
L7.jp_gdp 0.000000 0.000000 0.648 0.517
L7.jp_unemp -0.004673 0.013477 -0.347 0.729
L7.oil -0.000018 0.000202 -0.089 0.929
L7.usdjpy -0.000492 0.000958 -0.513 0.608
L7.jp_i -0.036287 0.037210 -0.975 0.329
L7.n225 -0.000003 0.000002 -1.283 0.200
L8.tokyo 0.000007 0.000012 0.572 0.567
L8.jp_gdp 0.000000 0.000000 0.285 0.776
L8.jp_unemp 0.008609 0.013433 0.641 0.522
L8.oil -0.000059 0.000203 -0.292 0.770
L8.usdjpy -0.001087 0.000961 -1.131 0.258
L8.jp_i -0.040442 0.037450 -1.080 0.280
L8.n225 -0.000001 0.000002 -0.271 0.786
L9.tokyo -0.000002 0.000012 -0.148 0.883
L9.jp_gdp -0.000000 0.000000 -1.145 0.252
L9.jp_unemp 0.019295 0.013425 1.437 0.151
L9.oil -0.000170 0.000204 -0.833 0.405
L9.usdjpy -0.002260 0.000959 -2.357 0.018
L9.jp_i 0.036886 0.037359 0.987 0.323
L9.n225 0.000004 0.000002 1.607 0.108
L10.tokyo -0.000011 0.000012 -0.927 0.354
L10.jp_gdp -0.000000 0.000000 -0.984 0.325
L10.jp_unemp -0.007735 0.013473 -0.574 0.566
L10.oil -0.000126 0.000204 -0.617 0.537
L10.usdjpy 0.000420 0.000966 0.435 0.664
L10.jp_i 0.067118 0.037678 1.781 0.075
L10.n225 0.000000 0.000002 0.106 0.916
L11.tokyo 0.000023 0.000012 1.867 0.062
L11.jp_gdp 0.000000 0.000000 0.028 0.978
L11.jp_unemp 0.000989 0.013469 0.073 0.941
L11.oil -0.000345 0.000202 -1.707 0.088
L11.usdjpy 0.001551 0.000960 1.615 0.106
L11.jp_i 0.000315 0.037678 0.008 0.993
L11.n225 -0.000005 0.000002 -2.431 0.015
L12.tokyo -0.000014 0.000012 -1.166 0.244
L12.jp_gdp -0.000000 0.000000 -1.169 0.242
L12.jp_unemp -0.000464 0.013459 -0.034 0.972
L12.oil 0.000127 0.000202 0.628 0.530
L12.usdjpy -0.000072 0.000965 -0.074 0.941
L12.jp_i 0.016560 0.037756 0.439 0.661
L12.n225 -0.000002 0.000002 -0.685 0.493
L13.tokyo -0.000005 0.000012 -0.395 0.693
L13.jp_gdp 0.000000 0.000000 0.447 0.655
L13.jp_unemp 0.017783 0.013459 1.321 0.186
L13.oil 0.000072 0.000200 0.362 0.717
L13.usdjpy 0.000384 0.000954 0.402 0.687
L13.jp_i 0.004112 0.037917 0.108 0.914
L13.n225 0.000002 0.000002 0.777 0.437
L14.tokyo -0.000026 0.000012 -2.162 0.031
L14.jp_gdp 0.000000 0.000000 0.222 0.824
L14.jp_unemp -0.036981 0.013573 -2.725 0.006
L14.oil 0.000184 0.000196 0.938 0.348
L14.usdjpy 0.000155 0.000945 0.164 0.869
L14.jp_i -0.066282 0.037862 -1.751 0.080
L14.n225 0.000001 0.000002 0.416 0.678
L15.tokyo -0.000008 0.000012 -0.663 0.507
L15.jp_gdp 0.000000 0.000000 0.297 0.766
L15.jp_unemp 0.002411 0.013601 0.177 0.859
L15.oil -0.000123 0.000180 -0.682 0.495
L15.usdjpy 0.000642 0.000924 0.695 0.487
L15.jp_i -0.007004 0.037644 -0.186 0.852
L15.n225 0.000000 0.000002 0.073 0.942
===============================================================================
Results for equation n225
===============================================================================
coefficient std. error t-stat prob
-------------------------------------------------------------------------------
const 10.783937 7.958958 1.355 0.175
L1.tokyo 0.108327 0.228790 0.473 0.636
L1.jp_gdp 0.008134 0.005292 1.537 0.124
L1.jp_unemp 59.384256 257.610801 0.231 0.818
L1.oil 13.685156 4.469679 3.062 0.002
L1.usdjpy 52.640040 18.168940 2.897 0.004
L1.jp_i -2160.429516 698.709019 -3.092 0.002
L1.n225 -0.023978 0.041243 -0.581 0.561
L2.tokyo -0.303374 0.228480 -1.328 0.184
L2.jp_gdp 0.005109 0.005287 0.966 0.334
L2.jp_unemp -69.094721 256.733489 -0.269 0.788
L2.oil 14.488216 4.415947 3.281 0.001
L2.usdjpy 13.923561 18.504029 0.752 0.452
L2.jp_i -494.675878 700.243155 -0.706 0.480
L2.n225 0.055777 0.041768 1.335 0.182
L3.tokyo -0.603774 0.228647 -2.641 0.008
L3.jp_gdp -0.010116 0.005255 -1.925 0.054
L3.jp_unemp 310.757400 256.745433 1.210 0.226
L3.oil 8.644184 4.060182 2.129 0.033
L3.usdjpy 0.487636 18.475337 0.026 0.979
L3.jp_i 726.126100 704.949349 1.030 0.303
L3.n225 0.017197 0.041857 0.411 0.681
L4.tokyo -0.161941 0.231980 -0.698 0.485
L4.jp_gdp -0.002818 0.005220 -0.540 0.589
L4.jp_unemp 43.657383 257.336541 0.170 0.865
L4.oil 9.373969 3.973729 2.359 0.018
L4.usdjpy 8.148560 18.379822 0.443 0.658
L4.jp_i 474.499466 707.855437 0.670 0.503
L4.n225 0.039663 0.041887 0.947 0.344
L5.tokyo 0.219571 0.231799 0.947 0.344
L5.jp_gdp -0.010153 0.005151 -1.971 0.049
L5.jp_unemp 32.473709 257.282258 0.126 0.900
L5.oil 2.475759 3.901425 0.635 0.526
L5.usdjpy -4.109719 18.480543 -0.222 0.824
L5.jp_i 156.671912 709.463970 0.221 0.825
L5.n225 -0.012168 0.042202 -0.288 0.773
L6.tokyo -0.225657 0.233164 -0.968 0.333
L6.jp_gdp 0.000866 0.005192 0.167 0.868
L6.jp_unemp -266.019302 257.914671 -1.031 0.302
L6.oil 3.083201 3.883020 0.794 0.427
L6.usdjpy -10.221159 18.467330 -0.553 0.580
L6.jp_i 1155.313525 710.037668 1.627 0.104
L6.n225 -0.047112 0.042551 -1.107 0.268
L7.tokyo 0.549556 0.233543 2.353 0.019
L7.jp_gdp -0.002749 0.005142 -0.535 0.593
L7.jp_unemp -44.033964 257.473150 -0.171 0.864
L7.oil -4.393678 3.856305 -1.139 0.255
L7.usdjpy 9.101730 18.304060 0.497 0.619
L7.jp_i -536.756282 710.885541 -0.755 0.450
L7.n225 -0.053393 0.042831 -1.247 0.213
L8.tokyo -0.015429 0.233052 -0.066 0.947
L8.jp_gdp 0.013767 0.005109 2.695 0.007
L8.jp_unemp -88.564279 256.639139 -0.345 0.730
L8.oil 4.831424 3.881351 1.245 0.213
L8.usdjpy 6.812512 18.358529 0.371 0.711
L8.jp_i -683.930916 715.459399 -0.956 0.339
L8.n225 -0.068080 0.043018 -1.583 0.114
L9.tokyo 0.280374 0.232310 1.207 0.227
L9.jp_gdp -0.013848 0.005045 -2.745 0.006
L9.jp_unemp -185.663043 256.479776 -0.724 0.469
L9.oil 6.733802 3.896201 1.728 0.084
L9.usdjpy 12.014858 18.315814 0.656 0.512
L9.jp_i 1965.663760 713.736977 2.754 0.006
L9.n225 -0.046882 0.042732 -1.097 0.273
L10.tokyo -0.028731 0.231923 -0.124 0.901
L10.jp_gdp -0.000551 0.004987 -0.111 0.912
L10.jp_unemp -101.278236 257.402649 -0.393 0.694
L10.oil -1.426188 3.894837 -0.366 0.714
L10.usdjpy -28.280815 18.458108 -1.532 0.125
L10.jp_i 1953.699449 719.826686 2.714 0.007
L10.n225 0.022146 0.043015 0.515 0.607
L11.tokyo 0.520901 0.231093 2.254 0.024
L11.jp_gdp 0.002547 0.004883 0.522 0.602
L11.jp_unemp 170.754318 257.328690 0.664 0.507
L11.oil -6.269520 3.862057 -1.623 0.105
L11.usdjpy 7.598052 18.348852 0.414 0.679
L11.jp_i -529.693089 719.818450 -0.736 0.462
L11.n225 -0.020526 0.042970 -0.478 0.633
L12.tokyo -0.107131 0.233777 -0.458 0.647
L12.jp_gdp -0.011385 0.004869 -2.338 0.019
L12.jp_unemp -341.823974 257.122585 -1.329 0.184
L12.oil -1.642383 3.855638 -0.426 0.670
L12.usdjpy 7.321627 18.436722 0.397 0.691
L12.jp_i 141.746139 721.315875 0.197 0.844
L12.n225 -0.001402 0.043348 -0.032 0.974
L13.tokyo 0.143531 0.232964 0.616 0.538
L13.jp_gdp -0.000897 0.004792 -0.187 0.851
L13.jp_unemp 109.576253 257.122747 0.426 0.670
L13.oil 3.726541 3.822281 0.975 0.330
L13.usdjpy -0.884042 18.230209 -0.048 0.961
L13.jp_i 778.532216 724.392554 1.075 0.282
L13.n225 0.051378 0.043295 1.187 0.235
L14.tokyo 0.229670 0.233203 0.985 0.325
L14.jp_gdp -0.003068 0.006000 -0.511 0.609
L14.jp_unemp 283.704127 259.312813 1.094 0.274
L14.oil 6.768652 3.738361 1.811 0.070
L14.usdjpy -37.076320 18.057659 -2.053 0.040
L14.jp_i 463.035285 723.335925 0.640 0.522
L14.n225 -0.010953 0.042996 -0.255 0.799
L15.tokyo 0.282975 0.234117 1.209 0.227
L15.jp_gdp -0.000973 0.006234 -0.156 0.876
L15.jp_unemp -202.843029 259.850802 -0.781 0.435
L15.oil 3.040779 3.437488 0.885 0.376
L15.usdjpy 13.924575 17.647338 0.789 0.430
L15.jp_i -96.019986 719.177157 -0.134 0.894
L15.n225 -0.035308 0.042159 -0.837 0.402
===============================================================================
Correlation matrix of residuals
tokyo jp_gdp jp_unemp oil usdjpy jp_i n225
tokyo 1.000000 0.007937 -0.089687 0.073427 0.111157 0.114727 0.393764
jp_gdp 0.007937 1.000000 0.076859 -0.018946 -0.028342 0.003233 0.076050
jp_unemp -0.089687 0.076859 1.000000 0.001843 -0.024381 0.032169 -0.081158
oil 0.073427 -0.018946 0.001843 1.000000 0.067905 0.024838 0.176879
usdjpy 0.111157 -0.028342 -0.024381 0.067905 1.000000 0.321144 0.390177
jp_i 0.114727 0.003233 0.032169 0.024838 0.321144 1.000000 0.263747
n225 0.393764 0.076050 -0.081158 0.176879 0.390177 0.263747 1.000000
# Calculate yhat
yhat_var2_6 = var2_model6.forecast(var_model6.endog, steps=len(test_diff6))
# clean up yhat
df_yhat_var2_6 = pd.DataFrame(yhat_var2_6)
df_yhat_var2_6.index = test_diff6.index
df_yhat_var2_6.columns = test_diff6.columns
df_yhat_var2_6
| tokyo | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| Date | |||||||
| 2021-01-04 | 9.709107 | -113.410223 | -0.001618 | 0.179858 | -0.139402 | -0.001512 | 78.150147 |
| 2021-01-05 | 8.741641 | 75.065053 | 0.003261 | 0.089964 | -0.054060 | -0.000822 | 38.893853 |
| 2021-01-06 | 2.219266 | -77.555580 | -0.008300 | 0.148149 | -0.066364 | 0.002384 | 33.240096 |
| 2021-01-07 | -10.834405 | -624.137319 | 0.000243 | 0.408428 | 0.000294 | 0.000256 | -38.528112 |
| 2021-01-08 | 2.062588 | -1095.489061 | 0.002165 | 0.280710 | 0.106668 | 0.002013 | 27.572715 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 2021-11-29 | -0.282191 | -0.951496 | 0.000110 | -0.003702 | -0.011049 | -0.000055 | 8.415911 |
| 2021-11-30 | -0.282192 | -0.951481 | 0.000110 | -0.003702 | -0.011049 | -0.000055 | 8.415910 |
| 2021-12-01 | -0.282191 | -0.951473 | 0.000110 | -0.003702 | -0.011049 | -0.000055 | 8.415906 |
| 2021-12-02 | -0.282191 | -0.951510 | 0.000110 | -0.003702 | -0.011049 | -0.000055 | 8.415908 |
| 2021-12-03 | -0.282191 | -0.951512 | 0.000110 | -0.003702 | -0.011049 | -0.000055 | 8.415908 |
226 rows × 7 columns
# Recover first order differencing
df_pred_var2_6 = pd.concat([tokyo_6[tokyo_6.index==pd.datetime(2020,12,30)], df_yhat_var2_6], axis=0).cumsum()
df_pred_var2_6 = df_pred_var2_6[df_pred_var2_6.index!=pd.datetime(2020,12,30)]
y_pred_var2_6 = df_pred_var2_6["tokyo"]
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
y_pred_var2_6.plot(ax=ax, label="VAR(15) with Oil, N225, Unemployement, GDP, USD-JPY", linestyle="dashed")
plt.legend()
plt.show()
var2_metrics6 = calc_metrics(y_test, y_pred_var2_6, return_dict=True)
Test Error Metrics MAE: 148.0882 MSE: 32951.911 MAPE: 0.0711 sMAPE: 0.0676 MASE: -6.363
y_train = tokyo_gas[tokyo_gas.index.year!=2021]
y_test = tokyo_gas[tokyo_gas.index.year==2021]
# Convert y_train into Prophet-suitable format
y_train_p = pd.DataFrame(y_train)
y_train_p = y_train_p.reset_index().rename(columns={"Tokyo Gas":"y", "Date":"ds"})
# Convert y_test into Prophet-suitable format
y_test_p = pd.DataFrame(y_test)
y_test_p = y_test_p.reset_index().rename(columns={"Tokyo Gas":"y", "Date":"ds"})
y_test_p.size
y_train_p.size
1950
from fbprophet.plot import plot_plotly
from fbprophet import Prophet
# Train model with default parameter
p_model_1 = Prophet()
p_model_1.fit(y_train_p)
INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
Initial log joint probability = -4.56358
<fbprophet.forecaster.Prophet at 0x7fe86cda90a0>
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
99 2748 0.0223508 667.786 1 1 120
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
199 2778.9 0.00948636 248.657 0.3866 1 231
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
299 2795.37 0.00321743 249.765 1 1 348
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
399 2804.08 0.00501663 280.05 1 1 465
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
499 2813.61 0.00319059 135.081 1 1 588
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
564 2818.53 0.000112124 213.617 5.2e-07 0.001 721 LS failed, Hessian reset
599 2822.4 0.000534906 113.517 0.7184 0.7184 761
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
699 2826.52 0.000647731 172.269 1 1 881
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
799 2829.72 0.00656521 436.282 0.4663 0.4663 997
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
803 2830.42 0.00074163 227.104 2.257e-06 0.001 1035 LS failed, Hessian reset
899 2833.93 0.00192072 132.104 0.2893 1 1148
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
996 2834.62 0.000100227 186.067 6.925e-07 0.001 1317 LS failed, Hessian reset
999 2834.64 0.000138779 109.043 1 1 1320
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1099 2834.97 0.00151381 114.104 1 1 1457
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1199 2836.65 0.000712144 179.669 0.2192 0.2192 1582
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1299 2837.25 0.000516239 82.9866 0.2651 0.6977 1702
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1399 2837.51 0.000653147 105.857 0.4754 0.4754 1826
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1499 2839.7 0.00734293 196.625 1 1 1949
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1599 2840.47 0.0025426 98.8108 1 1 2076
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1644 2840.61 2.15635e-05 66.5771 2.641e-07 0.001 2173 LS failed, Hessian reset
1678 2840.61 1.63198e-07 55.8203 0.2367 1 2217
Optimization terminated normally:
Convergence detected: relative gradient magnitude is below tolerance
# Create future_data dataframe
future_data = p_model_1.make_future_dataframe(periods=len(y_test_p), freq='D')
# Make prediction
p_result_1 = p_model_1.predict(future_data)
p_result_1.head(3)
| ds | trend | yhat_lower | yhat_upper | trend_lower | trend_upper | additive_terms | additive_terms_lower | additive_terms_upper | weekly | weekly_lower | weekly_upper | yearly | yearly_lower | yearly_upper | multiplicative_terms | multiplicative_terms_lower | multiplicative_terms_upper | yhat | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2017-01-04 | 2618.951503 | 2490.847581 | 2697.074230 | 2618.951503 | 2618.951503 | -22.638445 | -22.638445 | -22.638445 | -7.441870 | -7.441870 | -7.441870 | -15.196575 | -15.196575 | -15.196575 | 0.0 | 0.0 | 0.0 | 2596.313057 |
| 1 | 2017-01-05 | 2619.616665 | 2499.469519 | 2699.442227 | 2619.616665 | 2619.616665 | -23.379497 | -23.379497 | -23.379497 | -11.277116 | -11.277116 | -11.277116 | -12.102380 | -12.102380 | -12.102380 | 0.0 | 0.0 | 0.0 | 2596.237168 |
| 2 | 2017-01-06 | 2620.281827 | 2502.258062 | 2698.550801 | 2620.281827 | 2620.281827 | -22.732299 | -22.732299 | -22.732299 | -13.295097 | -13.295097 | -13.295097 | -9.437203 | -9.437203 | -9.437203 | 0.0 | 0.0 | 0.0 | 2597.549527 |
# Calculate predicted values (for test data)
p_pred_1 = p_result_1["yhat"][len(y_train):]
p_pred_1.index = y_test.index
# Display result
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
p_pred_1.plot(ax=ax, label="Prediction", linestyle="dashed")
ax.set_title("Prophet (default setting)")
plt.legend()
plt.show()
prophet1_metrics = calc_metrics(y_test, p_pred_1, return_dict=True)
Test Error Metrics MAE: 197.3705 MSE: 56943.7882 MAPE: 0.0943 sMAPE: 0.0886 MASE: -8.4805
# Add external regressor
y_train_p_toyko = y_train_p.copy()
y_train_p_toyko["oil"] = tokyo_gas_m["oil"].values[:len(y_train_p_toyko)]
y_train_p_toyko["n225"] = tokyo_gas_m["n225"].values[:len(y_train_p_toyko)]
# Train model with default parameter
p_model_2 = Prophet()
p_model_2.add_regressor("oil")
p_model_2.add_regressor("n225")
p_model_2.fit(y_train_p_toyko)
INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
<fbprophet.forecaster.Prophet at 0x7f31fec13410>
# Create future_data dataframe
future_data_toyko = p_model_2.make_future_dataframe(periods=len(y_test_p), freq='D')
future_data_toyko["oil"] = tokyo_gas_m["oil"].values
future_data_toyko["n225"] = tokyo_gas_m["n225"].values
future_data_toyko.head(2)
| ds | oil | n225 | |
|---|---|---|---|
| 0 | 2017-01-04 | 53.26 | 19594.160156 |
| 1 | 2017-01-05 | 53.77 | 19520.689453 |
# Make prediction
p_result_2 = p_model_2.predict(future_data_toyko)
p_result_2.head(3)
| ds | trend | yhat_lower | yhat_upper | trend_lower | trend_upper | additive_terms | additive_terms_lower | additive_terms_upper | extra_regressors_additive | ... | weekly | weekly_lower | weekly_upper | yearly | yearly_lower | yearly_upper | multiplicative_terms | multiplicative_terms_lower | multiplicative_terms_upper | yhat | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2017-01-04 | 2753.634345 | 2543.559076 | 2728.107631 | 2753.634345 | 2753.634345 | -116.943238 | -116.943238 | -116.943238 | -94.172335 | ... | 19.848219 | 19.848219 | 19.848219 | -42.619122 | -42.619122 | -42.619122 | 0.0 | 0.0 | 0.0 | 2636.691108 |
| 1 | 2017-01-05 | 2753.309951 | 2534.905638 | 2730.342785 | 2753.309951 | 2753.309951 | -123.723115 | -123.723115 | -123.723115 | -100.173035 | ... | 16.739797 | 16.739797 | 16.739797 | -40.289877 | -40.289877 | -40.289877 | 0.0 | 0.0 | 0.0 | 2629.586835 |
| 2 | 2017-01-06 | 2752.985556 | 2526.241706 | 2718.228527 | 2752.985556 | 2752.985556 | -127.609524 | -127.609524 | -127.609524 | -104.221791 | ... | 15.027839 | 15.027839 | 15.027839 | -38.415572 | -38.415572 | -38.415572 | 0.0 | 0.0 | 0.0 | 2625.376032 |
3 rows × 28 columns
# Calculate fitted values (for training data)
p_fitted_2 = p_result_2["yhat"][:len(y_train)]
p_fitted_2.index = y_train.index
# Calculate predicted values (for test data)
p_pred_2 = p_result_2["yhat"][len(y_train):]
p_pred_2.index = y_test.index
# Display result
fig, ax = plt.subplots(figsize=(16,5))
y_train.plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
p_pred_2.plot(ax=ax, label="Prediction", linestyle="dashed")
ax.set_title("Prophet with External Regressor")
plt.legend()
plt.show()
prophet2_metrics = calc_metrics(y_test, p_pred_2, return_dict=True)
Test Error Metrics MAE: 120.3906 MSE: 20752.0242 MAPE: 0.0569 sMAPE: 0.0551 MASE: -5.1729
df_train = tokyo_gas_weekly[:-23]
df_test = tokyo_gas_weekly[-23:]
# Convert y_train into Prophet-suitable format
y_train_p = pd.DataFrame(df_train)
y_train_p = y_train_p.reset_index().rename(columns={"Tokyo Gas":"y", "Date":"ds"})
# Convert y_test into Prophet-suitable format
y_test_p = pd.DataFrame(df_test)
y_test_p = y_test_p.reset_index().rename(columns={"Tokyo Gas":"y", "Date":"ds"})
y_train_p.head(2)
| ds | y | |
|---|---|---|
| 0 | 2017-01-08 | 2688.833333 |
| 1 | 2017-01-15 | 2638.250000 |
from fbprophet.plot import plot_plotly
from fbprophet import Prophet
# Train model with default parameter
prophet_uni = Prophet()
prophet_uni.fit(y_train_p)
INFO:fbprophet:Disabling weekly seasonality. Run prophet with weekly_seasonality=True to override this. INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
<fbprophet.forecaster.Prophet at 0x7f31feb230d0>
# Create future_data dataframe
future_data = prophet_uni.make_future_dataframe(periods=len(y_test_p), freq='W')
# Make prediction
result_prophet_uni = prophet_uni.predict(future_data)
result_prophet_uni.head(3)
| ds | trend | yhat_lower | yhat_upper | trend_lower | trend_upper | additive_terms | additive_terms_lower | additive_terms_upper | yearly | yearly_lower | yearly_upper | multiplicative_terms | multiplicative_terms_lower | multiplicative_terms_upper | yhat | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2017-01-08 | 2679.856082 | 2520.912654 | 2816.145145 | 2679.856082 | 2679.856082 | -10.684018 | -10.684018 | -10.684018 | -10.684018 | -10.684018 | -10.684018 | 0.0 | 0.0 | 0.0 | 2669.172064 |
| 1 | 2017-01-15 | 2682.312208 | 2536.257599 | 2831.972440 | 2682.312208 | 2682.312208 | -0.999257 | -0.999257 | -0.999257 | -0.999257 | -0.999257 | -0.999257 | 0.0 | 0.0 | 0.0 | 2681.312951 |
| 2 | 2017-01-22 | 2684.768334 | 2537.962647 | 2813.673938 | 2684.768334 | 2684.768334 | -18.771677 | -18.771677 | -18.771677 | -18.771677 | -18.771677 | -18.771677 | 0.0 | 0.0 | 0.0 | 2665.996658 |
df_train = weekly_m[:-23]
df_test = weekly_m[-23:]
# Convert y_train into Prophet-suitable format
y_train_p = pd.DataFrame(df_train)
y_train_p = y_train_p.reset_index().rename(columns={"Tokyo Gas":"y", "Date":"ds"})
# Convert y_test into Prophet-suitable format
y_test_p = pd.DataFrame(df_test)
y_test_p = y_test_p.reset_index().rename(columns={"Tokyo Gas":"y", "Date":"ds"})
def run_prophet_(_list, train_y_p=y_train_p, test_y=y_test, df_merged=weekly_m):
"""
Needs following inputs
- _list: column names used as external regressors
- train_y_p: y_train with prophet suitable format ('ds' and 'y')
- test_y: y_test wiht normal format (length and index are needed)
- df_merged: whole data (train+test) with external regressors
"""
# Create df with external regressor
y_train = train_y_p.copy()
for c in _list:
y_train[c] = df_merged[c].values[:len(y_train)]
# Train model with default parameter
prophet_model = Prophet()
for c in _list:
prophet_model.add_regressor(c)
prophet_model.fit(y_train)
# Create future_data dataframe
future_data = prophet_model.make_future_dataframe(periods=len(test_y), freq='W')
for c in _list:
future_data[c] = df_merged[c].values
# Make prediction
prophet_result = prophet_model.predict(future_data)
# Extract y_pred for test data
pred_prophet = prophet_result["yhat"][len(y_train):]
pred_prophet.index = test_y.index
# Return modle, result, predicted values
return prophet_model, prophet_result, pred_prophet
# With all regressors
prophet_all, result_prophet_all, pred_prophet_all = run_prophet_(_list=['jp_gdp', 'jp_unemp', 'oil', 'usdjpy', 'jp_i', 'n225'])
# Print result
print("Prophet (default) / all")
make_plot(y_train, y_test, pred_prophet_all, "Prophet (default) / all")
metrics_prophet_all = calc_metrics(y_test, pred_prophet_all, return_dict=True)
INFO:fbprophet:Disabling weekly seasonality. Run prophet with weekly_seasonality=True to override this. INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
Prophet (default) / all
Test Error Metrics MAE: 106.9358 MSE: 13783.4934 MAPE: 0.0516 sMAPE: 0.0532 MASE: -0.9018
# Run Prophet
prophet_n225, result_prophet_n225, pred_prophet_n225 = run_prophet_(_list=['n225'])
# Print result
print("Prophet (default) / n225")
make_plot(y_train, y_test, pred_prophet_n225, "Prophet (default) / n225")
metrics_prophet_n225 = calc_metrics(y_test, pred_prophet_n225, return_dict=True)
INFO:fbprophet:Disabling weekly seasonality. Run prophet with weekly_seasonality=True to override this. INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
Prophet (default) / n225
Test Error Metrics MAE: 111.7085 MSE: 18302.6546 MAPE: 0.0554 sMAPE: 0.0533 MASE: -0.9421
# Run Prophet
prophet_oilusdjpy, result_prophet_oilusdjpy, pred_prophet_oilusdjpy = run_prophet_(_list=['oil', 'usdjpy'])
# Print result
print("Prophet (default) / oil & USDJPY")
make_plot(y_train, y_test, pred_prophet_oilusdjpy, "Prophet (default) / oil & USDJPY")
metrics_prophet_oilusdjpy = calc_metrics(y_test, pred_prophet_oilusdjpy, return_dict=True)
INFO:fbprophet:Disabling weekly seasonality. Run prophet with weekly_seasonality=True to override this. INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
Prophet (default) / oil & USDJPY
Test Error Metrics MAE: 113.8575 MSE: 18311.7282 MAPE: 0.0565 sMAPE: 0.0543 MASE: -0.9602
# Run Prophet
prophet_gdpunemp, result_prophet_gdpunemp, pred_prophet_gdpunemp = run_prophet_(_list=['jp_gdp', 'jp_unemp'])
# Print result
print("Prophet (default) / GDP & Unemp")
make_plot(y_train, y_test, pred_prophet_gdpunemp, "Prophet (default) / GDP & Unemp")
metrics_prophet_gdpunemp = calc_metrics(y_test, pred_prophet_gdpunemp, return_dict=True)
INFO:fbprophet:Disabling weekly seasonality. Run prophet with weekly_seasonality=True to override this. INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
Prophet (default) / GDP & Unemp
Test Error Metrics MAE: 98.772 MSE: 12537.1325 MAPE: 0.0487 sMAPE: 0.0474 MASE: -0.833
df_tokyo = pd.Series(fitted_data, index = tokyo_gas.index)
df_tokyo = pd.DataFrame(df_tokyo).reset_index()
df_tokyo['ds'] = df_tokyo['Date']
df_tokyo = df_tokyo.drop(['Date'], axis = 1)
df_tokyo['y'] = df_tokyo[0]
df_tokyo = df_tokyo.drop([0], axis = 1)
train = df_tokyo.iloc[:int(len(df_tokyo)*0.75)]
test = df_tokyo.iloc[int(len(df_tokyo)*0.75):]
pmodel = Prophet()
pmodel.fit(train)
INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
Initial log joint probability = -16.8576
<fbprophet.forecaster.Prophet at 0x7fe86cda9e80>
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
99 1747.4 0.00611402 245.253 0.3515 1 125
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
152 1755.75 0.000645741 115.432 2.407e-06 0.001 233 LS failed, Hessian reset
199 1757.79 0.000744222 66.6534 10 1 296
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
299 1760.68 0.000414647 87.8548 0.9737 0.9737 427
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
367 1762.58 0.000476129 139.536 6.199e-06 0.001 547 LS failed, Hessian reset
399 1763.72 5.7492e-05 61.0545 0.2352 0.2352 585
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
427 1763.91 0.000326365 100.032 2.799e-06 0.001 667 LS failed, Hessian reset
485 1764.03 0.000532719 82.8289 6.901e-06 0.001 784 LS failed, Hessian reset
499 1764.05 8.24827e-06 66.2096 0.2261 0.2261 803
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
524 1764.06 4.71895e-05 69.5702 5.3e-07 0.001 886 LS failed, Hessian reset
549 1764.07 2.23388e-07 70.0129 0.2742 1 916
Optimization terminated normally:
Convergence detected: relative gradient magnitude is below tolerance
modeltuned = Prophet(changepoint_prior_scale=0.5, seasonality_prior_scale=0.01)
modeltuned.fit(train)
INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.
Initial log joint probability = -16.8576
<fbprophet.forecaster.Prophet at 0x7fe8cd030d00>
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
99 1923.63 0.0206014 348.146 1 1 123
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
199 2121.59 0.0102148 622.397 0.6409 0.6409 230
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
299 2139.48 0.0172466 202.801 1 1 347
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
399 2168.27 0.0327547 605.706 0.06411 1 457
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
499 2191.91 0.00680939 108.276 1 1 565
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
599 2200.09 0.0240781 199.207 1 1 672
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
699 2210.33 0.000585984 77.6127 1 1 784
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
799 2212.75 0.00891449 160.465 0.4812 1 890
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
899 2215.22 0.0202873 95.1593 1 1 1002
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
999 2219.94 0.00903165 108.131 1 1 1119
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1099 2225.69 0.00814579 54.3188 1 1 1229
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1199 2228.64 0.0102898 67.6323 1 1 1349
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1299 2230.32 0.00104789 30.7275 1 1 1463
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1399 2232.04 0.0164327 26.0938 1 1 1572
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1499 2232.59 0.000786256 16.9541 1 1 1685
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1599 2233.11 0.00202475 92.7398 0.5167 0.5167 1801
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1699 2233.31 0.0149236 64.4038 1 1 1919
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1703 2233.34 9.44778e-05 50.4146 3.28e-06 0.001 1965 LS failed, Hessian reset
1799 2233.59 0.00020531 37.6313 0.4755 0.4755 2075
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1822 2233.63 3.43194e-05 22.9839 7.485e-07 0.001 2147 LS failed, Hessian reset
1899 2233.78 0.00047106 11.6668 0.7269 0.7269 2230
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
1999 2234.31 0.051291 127.772 1 1 2343
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2099 2235.47 0.00133817 26.5849 1 1 2456
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2120 2235.51 4.71743e-05 26.5766 3.706e-06 0.001 2522 LS failed, Hessian reset
2199 2235.57 0.000739277 16.7006 1 1 2615
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2299 2235.9 0.00175838 41.3283 1 1 2734
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2399 2236.1 0.0166701 11.5143 1 1 2846
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2499 2236.25 0.000442497 53.7199 0.04056 1 2967
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2565 2236.3 1.67164e-05 13.4866 1.126e-06 0.001 3086 LS failed, Hessian reset
2599 2236.31 3.58674e-05 5.48813 1 1 3133
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2687 2236.44 0.000108085 78.4374 1.081e-06 0.001 3293 LS failed, Hessian reset
2699 2236.48 0.000214724 19.993 1 1 3305
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2769 2236.53 4.04011e-05 12.3104 3.377e-06 0.001 3424 LS failed, Hessian reset
2799 2236.53 6.03413e-05 11.8028 0.661 0.661 3463
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2823 2236.54 9.50755e-06 7.72469 1.082e-06 0.001 3537 LS failed, Hessian reset
2899 2236.55 5.7479e-05 7.98887 0.3813 1 3637
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
2999 2236.61 0.00412434 23.8031 0.6302 0.6302 3751
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
3099 2236.69 0.000147939 9.52879 0.5841 1 3866
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
3106 2236.69 1.35913e-05 10.5721 1.554e-06 0.001 3912 LS failed, Hessian reset
3199 2236.71 0.00103113 10.658 0.6586 0.6586 4026
Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes
3212 2236.71 1.00566e-05 7.93525 1.339e-06 0.001 4079 LS failed, Hessian reset
3232 2236.71 5.81152e-06 4.82531 1.038e-06 0.001 4148 LS failed, Hessian reset
3244 2236.71 5.86663e-06 4.3788 0.2883 1 4162
Optimization terminated normally:
Convergence detected: relative gradient magnitude is below tolerance
future_data = modeltuned.make_future_dataframe(periods=len(test), freq='d')
forecast_data = modeltuned.predict(future_data)
residual_pm = train['y'] - forecast_data['yhat'][:900]
residual_pm.plot()
plt.title('Prophet Residual Plot')
plt.show()
residual_pm.hist()
plt.title('Prophet Residual Histogram')
plt.show()
p = plot_acf(residual_pm, lags=40, title='Prophet Residual Autocorrelation Plot')
sm.stats.acorr_ljungbox(residual_pm, lags=[10], return_df=True, boxpierce=True)
| lb_stat | lb_pvalue | bp_stat | bp_pvalue | |
|---|---|---|---|---|
| 10 | 1586.427578 | 0.0 | 1578.159423 | 0.0 |
forecast_data_exp = pd.concat([
forecast_data['ds'],
inv_boxcox(forecast_data['yhat'], fitted_lambda)],axis=1)
tokyo_gas1 = pd.DataFrame(data = tokyo_gas).reset_index()
tokyo_gas1['ds'] = tokyo_gas1['Date']
tokyo_gas1 = tokyo_gas1.drop(['Date'], axis = 1)
tokyo_gas1['y'] = tokyo_gas1['Tokyo Gas']
tokyo_gas1 = tokyo_gas1.drop(['Tokyo Gas'], axis = 1)
tokyo_gas1['ds'] = pd.to_datetime(tokyo_gas1['ds'])
forecast_data_exp['ds'] = pd.to_datetime(forecast_data_exp['ds'])
comb = pd.merge_asof(forecast_data_exp[['ds','yhat']], tokyo_gas1[['ds','y']], on=['ds']).set_index('ds')
train_comb, test_comb = comb[comb.index<test['ds'].min()], comb[comb.index>=test['ds'].min()]
Prophet_TRAIN_MAE = mean_absolute_error(train_comb['y'], train_comb['yhat'])
Prophet_TRAIN_MSE = mean_squared_error(train_comb['y'], train_comb['yhat'])
Prophet_TRAIN_MAPE = mean_absolute_percentage_error(train_comb['y'], train_comb['yhat'], symmetric=False)
Prophet_TRAIN_SMAPE = mean_absolute_percentage_error(train_comb['y'], train_comb['yhat'])
print(f'Prophet TRAIN MAE: {Prophet_TRAIN_MAE}')
print(f'Prophet TRAIN MSE: {Prophet_TRAIN_MSE}')
print(f'Prophet TRAIN MAPE: {Prophet_TRAIN_MAPE}')
print(f'Prophet TRAIN SMAPE: {Prophet_TRAIN_SMAPE}')
Prophet TRAIN MAE: 48.14927362512074 Prophet TRAIN MSE: 4128.805761297893 Prophet TRAIN MAPE: 0.01820474440062813 Prophet TRAIN SMAPE: 0.01813276953161715
train_comb['y'].plot(color = 'blue', figsize=(15, 10))
test_comb['yhat'].plot(color='red')
test_comb['y'].plot(color = 'black')
plt.title('Propohet Model')
plt.show()
test_comb['yhat'].plot(color='red', figsize=(15, 10))
test_comb['y'].plot(color = 'black')
plt.title('Propohet Model Predictive Performance')
plt.show()
Prophet_TEST_MAE = mean_absolute_error(test_comb['y'], test_comb['yhat'])
Prophet_TEST_MSE = mean_squared_error(test_comb['y'], test_comb['yhat'])
Prophet_TEST_MAPE = mean_absolute_percentage_error(test_comb['y'], test_comb['yhat'], symmetric=False)
Prophet_TEST_SMAPE = mean_absolute_percentage_error(test_comb['y'], test_comb['yhat'])
/usr/local/lib/python3.7/dist-packages/sktime/performance_metrics/forecasting/_functions.py:1548: FutureWarning: In the percentage error metric functions the default argument symmetric=True is changing to symmetric=False in v0.12.0. FutureWarning, /usr/local/lib/python3.7/dist-packages/sktime/performance_metrics/forecasting/_functions.py:1548: FutureWarning: In the percentage error metric functions the default argument symmetric=True is changing to symmetric=False in v0.12.0. FutureWarning,
print(f'Prophet TEST MAE: {Prophet_TEST_MAE}')
print(f'Prophet TEST MSE: {Prophet_TEST_MSE}')
print(f'Prophet TEST MAPE: {Prophet_TEST_MAPE}')
print(f'Prophet TEST SMAPE: {Prophet_TEST_SMAPE}')
Prophet TEST MAE: 174.70018620167997 Prophet TEST MSE: 49763.55046425741 Prophet TEST MAPE: 0.07399878601699326 Prophet TEST SMAPE: 0.07858572220829031
fitted_data, fitted_lambda = stats.boxcox(tokyo_gas)
df_tokyo = pd.Series(fitted_data, index = tokyo_gas.index)
adf_kpss(df_tokyo)
stationarity from ADF test: False
stationarity from KPSS test('c'): False
stationarity from KPSS test('ct'): False
| ADF | KPSS('c') | KPSS('ct') | |
|---|---|---|---|
| Test Statistic | -1.541135 | 3.73534 | 0.701049 |
| p-value | 0.513073 | 0.01000 | 0.010000 |
| Lags Used | 8.000000 | 20.00000 | 20.000000 |
| Critical Value (1%) | -3.435848 | 0.73900 | 0.216000 |
| Critical Value (5%) | -2.863968 | 0.46300 | 0.146000 |
| Critical Value (10%) | -2.568063 | 0.34700 | 0.119000 |
y_train, y_test = temporal_train_test_split(df_tokyo)
y_train_nonseasdiff = y_train.diff(1).dropna()
main5 = df.copy().loc[df["SecuritiesCode"].isin([9501, 9503, 9531, 9532, 5020]), ["Date", "SecuritiesCode", "Close", "Name", "Category"]]
main5.loc[main5["Name"]=="ENEOS Holdings,Inc.", "Name"] = "ENEOS"
main5.loc[main5["Name"]=="Tokyo Electric Power Company Holdings,Incorporated", "Name"] = "Tokyo Electric Power"
main5.loc[main5["Name"]=="The Kansai Electric Power Company,Incorporated", "Name"] = "Kansai Electric Power"
main5.loc[main5["Name"]=="TOKYO GAS CO.,LTD.", "Name"] = "Tokyo Gas"
main5.loc[main5["Name"]=="OSAKA GAS CO.,LTD.", "Name"] = "Osaka Gas"
main5["Adjusted Close"] = main5["Close"]
main5.loc[(main5["SecuritiesCode"].isin([9531, 9532])) & (main5["Date"] < dt.datetime(2017,9,27)), "Adjusted Close"] = main5.loc[(main5["SecuritiesCode"].isin([9531, 9532])) & (main5["Date"] < dt.datetime(2017,9,27)), "Adjusted Close"]*5
tokyo_gas = main5.loc[main5["Name"]=="Tokyo Gas", "Adjusted Close"].dropna()
dataset = tokyo_gas[:900]
dataset = dataset.to_numpy()
def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
end_ix = i + n_steps
if end_ix > len(sequence)-1:
break
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return np.array(X), np.array(y)
n_steps = 30
X, y = split_sequence(dataset, n_steps)
n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))
X.shape
(870, 30, 1)
y.shape
(870,)
np.random.seed(42)
tf.random.set_seed(42)
model = keras.models.Sequential()
model.add(keras.layers.LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
model.add(keras.layers.LSTM(50, activation='relu'))
model.add(keras.layers.Dense(1, activation="relu"))
model.compile(optimizer='adam', loss='mse')
history = model.fit(X, y, epochs=50, verbose=0)
WARNING:tensorflow:Layer lstm_8 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
WARNING:tensorflow:Layer lstm_8 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
WARNING:tensorflow:Layer lstm_9 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
WARNING:tensorflow:Layer lstm_9 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
model.save("LSTM_model.h5")
keras.utils.plot_model(model, show_shapes=True)
pd.DataFrame(history.history).plot(figsize=(8, 5))
plt.grid(True)
plt.title('LSTM Learning Curve')
plt.show()
x_input = dataset[-30:]
x_input = x_input.reshape((1, n_steps, n_features))
x_input = dataset[-30:]
LSTM_pred = []
for i in range(0, 301):
x_input_arr = x_input.reshape((1, n_steps, n_features))
new_pred = model.predict(x_input_arr, verbose=0)
LSTM_pred.append(new_pred[0][0])
x_input = np.delete(x_input, 0)
x_input = np.append(x_input, new_pred)
LSTM_pred = pd.Series(LSTM_pred, index = tokyo_gas.index[-301:])
tokyo_gas[:900].plot(color = 'blue', figsize=(15, 10))
LSTM_pred.plot(color='red')
tokyo_gas[-301:].plot(color = 'black')
plt.title('LSTM Model')
plt.show()
LSTM_pred.plot(color='red', figsize=(15, 10))
tokyo_gas[-301:].plot(color = 'black')
plt.title('LSTM Model Predictive Performance')
plt.show()
LSTM_TEST_MAE = mean_absolute_error(tokyo_gas[-301:], LSTM_pred)
LSTM_TEST_MSE = mean_squared_error(tokyo_gas[-301:], LSTM_pred)
LSTM_TEST_MAPE = mean_absolute_percentage_error(tokyo_gas[-301:], LSTM_pred, symmetric=False)
LSTM_TEST_SMAPE = mean_absolute_percentage_error(tokyo_gas[-301:], LSTM_pred)
/usr/local/lib/python3.7/dist-packages/sktime/performance_metrics/forecasting/_functions.py:1548: FutureWarning: In the percentage error metric functions the default argument symmetric=True is changing to symmetric=False in v0.12.0. FutureWarning, /usr/local/lib/python3.7/dist-packages/sktime/performance_metrics/forecasting/_functions.py:1548: FutureWarning: In the percentage error metric functions the default argument symmetric=True is changing to symmetric=False in v0.12.0. FutureWarning,
print(f'LSTM TEST MAE: {LSTM_TEST_MAE}')
print(f'LSTM TEST MSE: {LSTM_TEST_MSE}')
print(f'LSTM TEST MAPE: {LSTM_TEST_MAPE}')
print(f'LSTM TEST SMAPE: {LSTM_TEST_SMAPE}')
LSTM TEST MAE: 88.87023723006645 LSTM TEST MSE: 14273.594674552101 LSTM TEST MAPE: 0.03885517634287059 LSTM TEST SMAPE: 0.03966203776120834
# Take average of each week
eneos_weekly = eneos.resample("W").mean()
# NA value in 2019-05-05 due to consecutive national holidays (called "Golden Week")
eneos_weekly[eneos_weekly.isna()]
Date 2019-05-05 NaN Freq: W-SUN, Name: eneos, dtype: float64
# Fill NA with interpolation
eneos_weekly = eneos_weekly.interpolate()
# Check stationarity
create_plot_adf_kpss(eneos_weekly, "ENEOS_weekly")
ADF/KPSS tests for ENEOS_weekly (alpha=0.05):
stationarity from ADF test: False
stationarity from KPSS test('c'): False
stationarity from KPSS test('ct'): False
| ADF | KPSS('c') | KPSS('ct') | |
|---|---|---|---|
| Test Statistic | -1.402885 | 1.17686 | 0.264062 |
| p-value | 0.580892 | 0.01000 | 0.010000 |
| Lags Used | 1.000000 | 10.00000 | 10.000000 |
| Critical Value (1%) | -3.456257 | 0.73900 | 0.216000 |
| Critical Value (5%) | -2.872942 | 0.46300 | 0.146000 |
| Critical Value (10%) | -2.572846 | 0.34700 | 0.119000 |
# Check seasonality: No seasonality
df_season = pd.DataFrame(eneos_weekly)
df_season["year"] = df_season.index.year
df_season["week"] = df_season.index.week
# Line plot
fig, ax = plt.subplots(figsize=(15,6))
sns.lineplot("week", "eneos", data=df_season, hue="year", ax=ax, legend=None)
ax.grid()
plt.show()
# Check stationarity for differenced data
create_plot_adf_kpss(eneos_weekly.diff().dropna(), "ENEOS_weekly_diff")
ADF/KPSS tests for ENEOS_weekly_diff (alpha=0.05):
stationarity from ADF test: True
stationarity from KPSS test('c'): True
stationarity from KPSS test('ct'): True
| ADF | KPSS('c') | KPSS('ct') | |
|---|---|---|---|
| Test Statistic | -1.243787e+01 | 0.138115 | 0.098419 |
| p-value | 3.811269e-23 | 0.100000 | 0.100000 |
| Lags Used | 0.000000e+00 | 3.000000 | 3.000000 |
| Critical Value (1%) | -3.456257e+00 | 0.739000 | 0.216000 |
| Critical Value (5%) | -2.872942e+00 | 0.463000 | 0.146000 |
| Critical Value (10%) | -2.572846e+00 | 0.347000 | 0.119000 |
plot_acf_pacf(eneos_weekly.diff().dropna(), "ENEOS Weekly (diff(1))", lags=80)
# GDP
gdp_weekly = gdp_s.resample("W").mean().fillna(method="ffill")
# unemplyment rate
unemp_weekly = unemp_s.resample("W").mean().fillna(method="ffill")
# Oil Price
oil_weekly = oil_s.resample("W").mean()
# JPY/USD
usdjpy_weekly = usdjpy_s.resample("W").mean()
# Interest Rate (long-term)
interest_weekly = interest_s.resample("W").mean()
# Find NAs for interest_weekly
interest_weekly[interest_weekly.isna()]
Date 2019-05-05 NaN Freq: W-SUN, Name: jp_i, dtype: float64
# Interpolation
interest_weekly = interest_weekly.interpolate()
# Nikkei 225 index
n225_weekly = n225_s.resample("W").mean()
# Find NAs
n225_weekly[n225_weekly.isna()]
Date 2019-05-05 NaN Freq: W-SUN, Name: n225, dtype: float64
# Interpolation
n225_weekly = n225_weekly.interpolate()
# Merge data
weekly_m = pd.concat([eneos_weekly, gdp_weekly, unemp_weekly, oil_weekly, usdjpy_weekly, interest_weekly, n225_weekly], axis=1)
# Show rows with NAs => Need to fill jp_gdp (since last observation for gdp_weekly is for 2021-10-03)
weekly_m[weekly_m.isnull().any(axis=1)]
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| Date | |||||||
| 2017-01-01 | NaN | 547306.2 | 3.1 | NaN | NaN | NaN | NaN |
| 2021-10-10 | 454.200 | NaN | 2.8 | 78.504000 | 111.295401 | 0.07200 | 27904.605859 |
| 2021-10-17 | 460.300 | NaN | 2.8 | 81.176000 | 113.262399 | 0.08740 | 28497.729687 |
| 2021-10-24 | 467.280 | NaN | 2.8 | 83.476000 | 114.278200 | 0.09280 | 29001.992188 |
| 2021-10-31 | 464.500 | NaN | 2.8 | 83.844000 | 113.801801 | 0.09940 | 28903.487891 |
| 2021-11-07 | 462.025 | NaN | 2.9 | 81.788000 | 114.039799 | 0.08300 | 29643.479980 |
| 2021-11-14 | 459.300 | NaN | 2.9 | 81.930000 | 113.517799 | 0.06840 | 29357.424219 |
| 2021-11-21 | 443.880 | NaN | 2.9 | 78.992000 | 114.297801 | 0.08020 | 29723.555859 |
| 2021-11-28 | 436.075 | NaN | 2.9 | 77.793333 | 114.909200 | 0.08475 | 29331.916992 |
| 2021-12-05 | 425.140 | NaN | 2.8 | 66.890000 | 113.345203 | 0.06980 | 27964.847656 |
| 2021-12-12 | NaN | NaN | NaN | 71.314000 | 113.431400 | 0.05800 | 28481.365625 |
| 2021-12-19 | NaN | NaN | NaN | 71.184000 | 113.724001 | 0.05820 | 28628.970313 |
| 2021-12-26 | NaN | NaN | NaN | 71.625000 | 113.950601 | 0.06920 | 28519.714062 |
| 2022-01-02 | NaN | NaN | NaN | 76.048000 | 114.781801 | 0.07950 | 28861.053223 |
# Drop rows when "eneos" is NA
weekly_m = weekly_m.dropna(subset=["eneos"])
# Show rows with NAs
weekly_m[weekly_m.isnull().any(axis=1)]
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| Date | |||||||
| 2021-10-10 | 454.200 | NaN | 2.8 | 78.504000 | 111.295401 | 0.07200 | 27904.605859 |
| 2021-10-17 | 460.300 | NaN | 2.8 | 81.176000 | 113.262399 | 0.08740 | 28497.729687 |
| 2021-10-24 | 467.280 | NaN | 2.8 | 83.476000 | 114.278200 | 0.09280 | 29001.992188 |
| 2021-10-31 | 464.500 | NaN | 2.8 | 83.844000 | 113.801801 | 0.09940 | 28903.487891 |
| 2021-11-07 | 462.025 | NaN | 2.9 | 81.788000 | 114.039799 | 0.08300 | 29643.479980 |
| 2021-11-14 | 459.300 | NaN | 2.9 | 81.930000 | 113.517799 | 0.06840 | 29357.424219 |
| 2021-11-21 | 443.880 | NaN | 2.9 | 78.992000 | 114.297801 | 0.08020 | 29723.555859 |
| 2021-11-28 | 436.075 | NaN | 2.9 | 77.793333 | 114.909200 | 0.08475 | 29331.916992 |
| 2021-12-05 | 425.140 | NaN | 2.8 | 66.890000 | 113.345203 | 0.06980 | 27964.847656 |
# Fill NAs for jp_gdp with forward fill
weekly_m["jp_gdp"] = weekly_m["jp_gdp"].fillna(method="ffill")
# Show rows with NAs
weekly_m[weekly_m.isnull().any(axis=1)]
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| Date |
# Plot external regressors
fig, axes = plt.subplots(2,3, figsize=(16,8), tight_layout=True)
axes_flat = [s for sublist in axes for s in sublist]
for i, c in enumerate(weekly_m.columns.difference(["eneos"])):
weekly_m[c].plot(ax=axes_flat[i])
axes_flat[i].set_title(c)
plt.show()
# Using data before June 2021 as training
df_train = weekly_m[:-23]
df_test = weekly_m[-23:]
df_train
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| Date | |||||||
| 2017-01-08 | 510.933333 | 547306.2 | 3.1 | 53.3425 | 116.865399 | 0.058333 | 19523.059896 |
| 2017-01-15 | 514.000000 | 547306.2 | 3.1 | 52.0660 | 115.753801 | 0.052250 | 19272.021973 |
| 2017-01-22 | 515.660000 | 547306.2 | 3.1 | 51.8225 | 114.199800 | 0.059200 | 19002.659766 |
| 2017-01-29 | 526.200000 | 547306.2 | 3.1 | 52.7420 | 113.635599 | 0.068200 | 19121.262109 |
| 2017-02-05 | 531.400000 | 547306.2 | 3.1 | 53.3280 | 113.432199 | 0.096200 | 19078.209766 |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 2021-05-30 | 453.560000 | 544693.4 | 3.1 | 66.3980 | 109.110199 | 0.081200 | 28651.839844 |
| 2021-06-06 | 462.780000 | 544693.4 | 3.1 | 68.7425 | 109.750798 | 0.087000 | 28924.037891 |
| 2021-06-13 | 470.080000 | 544693.4 | 3.1 | 70.1120 | 109.475800 | 0.067000 | 28950.178516 |
| 2021-06-20 | 470.500000 | 544693.4 | 3.1 | 71.5460 | 110.185799 | 0.058600 | 29175.304297 |
| 2021-06-27 | 466.940000 | 544693.4 | 3.1 | 73.4840 | 110.632399 | 0.060400 | 28742.272266 |
234 rows × 7 columns
df_test
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| Date | |||||||
| 2021-07-04 | 468.480000 | 538923.3 | 2.9 | 74.068000 | 110.931200 | 0.060800 | 28828.495312 |
| 2021-07-11 | 468.260000 | 538923.3 | 2.9 | 73.345000 | 110.617801 | 0.033000 | 28333.359766 |
| 2021-07-18 | 472.900000 | 538923.3 | 2.9 | 73.188000 | 110.179399 | 0.020600 | 28435.583984 |
| 2021-07-25 | 453.900000 | 538923.3 | 2.9 | 69.684000 | 109.944598 | 0.015667 | 27529.633464 |
| 2021-08-01 | 460.400000 | 538923.3 | 2.9 | 72.750000 | 109.990399 | 0.019000 | 27690.235938 |
| 2021-08-08 | 464.960000 | 538923.3 | 2.9 | 69.500000 | 109.449399 | 0.014000 | 27711.017578 |
| 2021-08-15 | 466.850000 | 538923.3 | 2.9 | 68.334000 | 110.388000 | 0.032250 | 27987.707520 |
| 2021-08-22 | 431.860000 | 538923.3 | 2.9 | 65.048000 | 109.613600 | 0.015800 | 27365.598047 |
| 2021-08-29 | 421.980000 | 538923.3 | 2.9 | 67.590000 | 109.854399 | 0.025800 | 27666.914062 |
| 2021-09-05 | 430.480000 | 538923.3 | 2.9 | 69.150000 | 109.981200 | 0.035200 | 28400.293359 |
| 2021-09-12 | 446.980000 | 538923.3 | 2.9 | 68.982500 | 109.975800 | 0.049600 | 30029.454297 |
| 2021-09-19 | 460.880000 | 538923.3 | 2.9 | 71.688000 | 109.742001 | 0.050400 | 30490.514062 |
| 2021-09-26 | 460.166667 | 538923.3 | 2.9 | 72.180000 | 109.740999 | 0.051333 | 29909.307292 |
| 2021-10-03 | 462.480000 | 540745.2 | 2.8 | 75.454000 | 111.330598 | 0.074400 | 29638.408203 |
| 2021-10-10 | 454.200000 | 540745.2 | 2.8 | 78.504000 | 111.295401 | 0.072000 | 27904.605859 |
| 2021-10-17 | 460.300000 | 540745.2 | 2.8 | 81.176000 | 113.262399 | 0.087400 | 28497.729687 |
| 2021-10-24 | 467.280000 | 540745.2 | 2.8 | 83.476000 | 114.278200 | 0.092800 | 29001.992188 |
| 2021-10-31 | 464.500000 | 540745.2 | 2.8 | 83.844000 | 113.801801 | 0.099400 | 28903.487891 |
| 2021-11-07 | 462.025000 | 540745.2 | 2.9 | 81.788000 | 114.039799 | 0.083000 | 29643.479980 |
| 2021-11-14 | 459.300000 | 540745.2 | 2.9 | 81.930000 | 113.517799 | 0.068400 | 29357.424219 |
| 2021-11-21 | 443.880000 | 540745.2 | 2.9 | 78.992000 | 114.297801 | 0.080200 | 29723.555859 |
| 2021-11-28 | 436.075000 | 540745.2 | 2.9 | 77.793333 | 114.909200 | 0.084750 | 29331.916992 |
| 2021-12-05 | 425.140000 | 540745.2 | 2.8 | 66.890000 | 113.345203 | 0.069800 | 27964.847656 |
# extract eneos sereis as y_train and y_test
y_train = df_train["eneos"]
y_test = df_test["eneos"]
# Make prediction with arima model from pmdarima
def predict_from_arima(arima_model, test_y, test_X=None):
y_pred = arima_model.predict(n_periods=len(test_y), X=test_X)
y_pred = pd.Series(y_pred)
y_pred.index = test_y.index
return y_pred
# Make plot fro training/test/prediction
def make_plot(train_y, test_y, pred_y, title):
fig, ax = plt.subplots(figsize=(16,5))
train_y.plot(ax=ax, label="Training")
test_y.plot(ax=ax, label="Test", color="black")
pred_y.plot(ax=ax, label=title, linestyle="dashed")
plt.legend()
plt.show()
# Function to run rolling fit
def rolling_arima3(train_y_raw, test_y_raw):
# Series to store predicted values
y_pred = pd.Series(name="eneos")
# Set initial training / test data / prediction
train_y = train_y_raw.copy()
test_y = test_y_raw.copy()
for i in range(len(test_y)):
# Train ARIMA and make 1-step ahead forecast
arima = auto_arima(train_y, start_p=0, d=1, start_q=0, seasonal=False)
arima_order = arima.get_params()["order"]
ypred = pd.Series(arima.predict(1), name="eneos", index=[test_y_raw.index[i]])
# Store predicted value into y_pred
y_pred = y_pred.append(ypred)
# Add 1-step ahead true observation to train_y and drop first observation from train_y
train_y = train_y.append(test_y[i:i+1])[1:]
# Print progress
print(f"Index {i+1}: ARIMA{str(arima_order)} : Prediction: {ypred.index[0].strftime('%Y-%m-%d')} {ypred.values}")
return y_pred
pred_airma_rolling_uni3 = rolling_arima3(y_train, y_test)
Index 1: ARIMA(0, 1, 1) : Prediction: 2021-07-04 [466.04885912] Index 2: ARIMA(0, 1, 1) : Prediction: 2021-07-11 [469.12423017] Index 3: ARIMA(0, 1, 1) : Prediction: 2021-07-18 [468.03096663] Index 4: ARIMA(0, 1, 1) : Prediction: 2021-07-25 [474.18261045] Index 5: ARIMA(0, 1, 1) : Prediction: 2021-08-01 [448.54126388] Index 6: ARIMA(0, 1, 1) : Prediction: 2021-08-08 [463.53587825] Index 7: ARIMA(0, 1, 1) : Prediction: 2021-08-15 [465.33062447] Index 8: ARIMA(0, 1, 1) : Prediction: 2021-08-22 [467.25361844] Index 9: ARIMA(0, 1, 1) : Prediction: 2021-08-29 [422.5013441] Index 10: ARIMA(0, 1, 1) : Prediction: 2021-09-05 [421.83617573] Index 11: ARIMA(0, 1, 1) : Prediction: 2021-09-12 [432.76519871] Index 12: ARIMA(0, 1, 1) : Prediction: 2021-09-19 [450.75625216] Index 13: ARIMA(0, 1, 1) : Prediction: 2021-09-26 [463.58074917] Index 14: ARIMA(0, 1, 1) : Prediction: 2021-10-03 [459.25553856] Index 15: ARIMA(0, 1, 1) : Prediction: 2021-10-10 [463.32435339] Index 16: ARIMA(0, 1, 1) : Prediction: 2021-10-17 [451.7806552] Index 17: ARIMA(0, 1, 1) : Prediction: 2021-10-24 [462.54864079] Index 18: ARIMA(0, 1, 1) : Prediction: 2021-10-31 [468.53173626] Index 19: ARIMA(0, 1, 1) : Prediction: 2021-11-07 [463.42066449] Index 20: ARIMA(0, 1, 1) : Prediction: 2021-11-14 [461.65189994] Index 21: ARIMA(0, 1, 1) : Prediction: 2021-11-21 [458.67311245] Index 22: ARIMA(0, 1, 1) : Prediction: 2021-11-28 [439.93671082] Index 23: ARIMA(0, 1, 1) : Prediction: 2021-12-05 [435.04550794]
print("Rolling ARIMA3 / No-exog")
make_plot(y_train, y_test, pred_airma_rolling_uni3, "Rolling ARIMA3 / No-exog")
metrics_airma_rolling_uni3 = calc_metrics(y_test, pred_airma_rolling_uni3, return_dict=True)
Rolling ARIMA3 / No-exog
Test Error Metrics MAE: 7.7174 MSE: 120.645 MAPE: 0.0172 sMAPE: 0.0171 MASE: 10.232
# Function to run rolling fit
def rolling_arima4(train_y_raw, test_y_raw):
# Series to store predicted values
y_pred = pd.Series(name="eneos")
# Set initial training / test data / prediction
train_y = train_y_raw.copy()[-52:]
test_y = test_y_raw.copy()
for i in range(len(test_y)):
# Train ARIMA and make 1-step ahead forecast
arima = auto_arima(train_y, start_p=0, d=1, start_q=0, seasonal=False)
arima_order = arima.get_params()["order"]
ypred = pd.Series(arima.predict(1), name="eneos", index=[test_y_raw.index[i]])
# Store predicted value into y_pred
y_pred = y_pred.append(ypred)
# Add 1-step ahead true observation to train_y and drop first observation from train_y
train_y = train_y.append(test_y[i:i+1])[1:]
# Print progress
print(f"Index {i+1}: ARIMA{str(arima_order)} : Prediction: {ypred.index[0].strftime('%Y-%m-%d')} {ypred.values}")
return y_pred
pred_airma_rolling_uni4 = rolling_arima4(y_train, y_test)
Index 1: ARIMA(0, 1, 1) : Prediction: 2021-07-04 [465.92946759] Index 2: ARIMA(0, 1, 1) : Prediction: 2021-07-11 [469.25236729] Index 3: ARIMA(0, 1, 1) : Prediction: 2021-07-18 [467.96385521] Index 4: ARIMA(1, 1, 0) : Prediction: 2021-07-25 [474.3483901] Index 5: ARIMA(1, 1, 0) : Prediction: 2021-08-01 [448.3256459] Index 6: ARIMA(1, 1, 0) : Prediction: 2021-08-08 [462.49101024] Index 7: ARIMA(1, 1, 0) : Prediction: 2021-08-15 [466.35811465] Index 8: ARIMA(1, 1, 0) : Prediction: 2021-08-22 [467.42946451] Index 9: ARIMA(0, 1, 0) : Prediction: 2021-08-29 [431.86] Index 10: ARIMA(1, 1, 0) : Prediction: 2021-09-05 [419.0842712] Index 11: ARIMA(0, 1, 1) : Prediction: 2021-09-12 [432.80907358] Index 12: ARIMA(0, 1, 1) : Prediction: 2021-09-19 [450.94037173] Index 13: ARIMA(1, 1, 0) : Prediction: 2021-09-26 [465.10402167] Index 14: ARIMA(0, 1, 1) : Prediction: 2021-10-03 [459.12547473] Index 15: ARIMA(0, 1, 1) : Prediction: 2021-10-10 [463.45498001] Index 16: ARIMA(0, 1, 1) : Prediction: 2021-10-17 [451.56225879] Index 17: ARIMA(0, 1, 1) : Prediction: 2021-10-24 [462.65304672] Index 18: ARIMA(0, 1, 1) : Prediction: 2021-10-31 [468.54404857] Index 19: ARIMA(0, 1, 1) : Prediction: 2021-11-07 [463.38565373] Index 20: ARIMA(0, 1, 1) : Prediction: 2021-11-14 [461.65000441] Index 21: ARIMA(0, 1, 1) : Prediction: 2021-11-21 [458.64814947] Index 22: ARIMA(0, 1, 1) : Prediction: 2021-11-28 [439.57177806] Index 23: ARIMA(1, 1, 0) : Prediction: 2021-12-05 [433.7105553]
print("Rolling ARIMA4 / No-exog")
make_plot(y_train, y_test, pred_airma_rolling_uni4, "Rolling ARIMA4 / No-exog")
metrics_airma_rolling_uni4 = calc_metrics(y_test, pred_airma_rolling_uni4, return_dict=True)
Rolling ARIMA4 / No-exog
Test Error Metrics MAE: 8.2791 MSE: 127.8672 MAPE: 0.0185 sMAPE: 0.0184 MASE: 10.9768
# Function to run rolling fit
def rolling_arimax3(exog_list, train_df=df_train, test_df=df_test, verbose=False):
"""
Input:
(1) exog_list: column names for external regressors
(2) train_df: df_train
(3) test_df: df for extracting index / exog for training
Process:
(1) Build model using auto_arima() and whole training data (eneos + external regressors)
(2) Make one-step ahead prediction for external regressors using actual values available at the point of prediction
(3) Use model from (1) and external regressors from (2) to make one-step ahead prediction for eneos
(4) Store predicted eneos and predicted external regressors as training data
(5) Repeat (1) through (4) over test data periods
Outoput:
- Prediction for Eneos (23-step ahead)
"""
# Extract eneos from train/test df as train_y and test_y
train_y = train_df.copy()["eneos"]
test_y = test_df.copy()["eneos"]
# Create df for external regressors (Use whole trainig data)
df_X = train_df.copy()[exog_list]
# Series to store predicted values
y_pred = pd.Series(name="eneos")
for i in range(test_df.shape[0]):
# Create ARIMAX model using training eneos and external regressors (df_X)
arimax = auto_arima(train_y, X=df_X, start_p=0, d=1, start_q=0, seasonal=False)
arimax_order = arimax.get_params()["order"]
# Create df for predicted external regressors
df_pred_exog = pd.DataFrame(columns=exog_list, index=[test_df.index[i]])
# Make one-step ahead forecast for external regressors
for c in exog_list:
arima_exog = auto_arima(df_X[c], start_p=0, d=1, start_q=0, seasonal=False)
pred_exog = arima_exog.predict(n_periods=1)
df_pred_exog[c] = pred_exog
# Make prediction
ypred = arimax.predict(n_periods=1, X=df_pred_exog)
ypred = pd.Series(ypred, name="eneos", index=[test_df.index[i]])
# Store predicted value into y_pred
y_pred = y_pred.append(ypred)
# Add 1-step ahead true observation to train_y and drop first observation from train_y
train_y = train_y.append(test_y[i:i+1])[1:]
# Add 1-step ahead true external regressors into df_X and drop first observation from df_X
df_X = df_X.append(test_df.copy().iloc[i][exog_list])[1:]
# Print progress
if verbose:
print(f"Index {i + 1}")
print(f" ARIMAX{str(arimax_order)} : Prediction: {ypred.index[0].strftime('%Y-%m-%d')} {ypred.values}")
print(" One-step ahead forecast for External Regressors:")
display(df_pred_exog)
return y_pred
pred_rolling_arimax_all3 = rolling_arimax3(exog_list=['jp_gdp', 'jp_unemp', 'oil', 'usdjpy', 'jp_i', 'n225'], verbose=True)
Index 1
ARIMAX(0, 1, 1) : Prediction: 2021-07-04 [465.91477323]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-07-04 | 544693.4 | 3.1 | 73.904943 | 110.552615 | 0.060778 | 28651.033847 |
Index 2
ARIMAX(0, 1, 1) : Prediction: 2021-07-11 [467.89726752]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-07-11 | 538923.3 | 2.9 | 74.110167 | 110.879991 | 0.060803 | 28858.915501 |
Index 3
ARIMAX(0, 1, 1) : Prediction: 2021-07-18 [467.33418279]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-07-18 | 538923.3 | 2.9 | 73.147407 | 110.534072 | 0.029448 | 28225.604976 |
Index 4
ARIMAX(0, 1, 1) : Prediction: 2021-07-25 [474.06311931]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-07-25 | 538923.3 | 2.9 | 73.198394 | 110.197102 | 0.019372 | 28475.57012 |
Index 5
ARIMAX(0, 1, 1) : Prediction: 2021-08-01 [450.19378469]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-01 | 538923.3 | 2.9 | 68.778817 | 110.00473 | 0.015115 | 27367.662905 |
Index 6
ARIMAX(0, 1, 1) : Prediction: 2021-08-08 [463.17176243]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-08 | 538923.3 | 2.9 | 73.730575 | 110.040719 | 0.019578 | 27759.97867 |
Index 7
ARIMAX(0, 1, 1) : Prediction: 2021-08-15 [464.46909374]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-15 | 538923.3 | 2.9 | 68.513237 | 109.380316 | 0.013173 | 27703.860008 |
Index 8
ARIMAX(0, 1, 1) : Prediction: 2021-08-22 [467.44840949]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-22 | 538923.3 | 2.9 | 68.291897 | 110.592858 | 0.03225 | 28039.377361 |
Index 9
ARIMAX(0, 1, 1) : Prediction: 2021-08-29 [425.91171047]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-29 | 538923.3 | 2.9 | 64.286315 | 109.358079 | 0.0158 | 27256.389598 |
Index 10
ARIMAX(0, 1, 1) : Prediction: 2021-09-05 [422.51938531]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-09-05 | 538923.3 | 2.9 | 68.327579 | 110.015825 | 0.0258 | 27748.853309 |
Index 11
ARIMAX(0, 1, 1) : Prediction: 2021-09-12 [431.77977365]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-09-12 | 538923.3 | 2.9 | 69.335438 | 109.951308 | 0.0352 | 28506.469404 |
Index 12
ARIMAX(0, 1, 1) : Prediction: 2021-09-19 [449.87442377]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-09-19 | 538923.3 | 2.9 | 68.902585 | 109.952471 | 0.0496 | 30312.484859 |
Index 13
ARIMAX(0, 1, 1) : Prediction: 2021-09-26 [463.03152018]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-09-26 | 538923.3 | 2.9 | 72.31478 | 109.722053 | 0.0504 | 30524.521575 |
Index 14
ARIMAX(0, 1, 1) : Prediction: 2021-10-03 [459.45598211]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-03 | 538923.3 | 2.9 | 72.151752 | 109.784679 | 0.051333 | 29792.865017 |
Index 15
ARIMAX(0, 1, 1) : Prediction: 2021-10-10 [462.17379157]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-10 | 540745.2 | 2.8 | 76.185142 | 111.464668 | 0.0744 | 29589.177257 |
Index 16
ARIMAX(0, 1, 1) : Prediction: 2021-10-17 [450.42288851]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-17 | 540745.2 | 2.8 | 79.192979 | 111.012492 | 0.072 | 27509.708505 |
Index 17
ARIMAX(0, 1, 1) : Prediction: 2021-10-24 [461.2741628]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-24 | 540745.2 | 2.8 | 81.784506 | 113.415063 | 0.0874 | 28604.580802 |
Index 18
ARIMAX(0, 1, 1) : Prediction: 2021-10-31 [467.31047581]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-31 | 540745.2 | 2.8 | 84.005536 | 113.945806 | 0.0928 | 29065.758955 |
Index 19
ARIMAX(0, 1, 1) : Prediction: 2021-11-07 [464.17870306]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-11-07 | 540745.2 | 2.8 | 83.928965 | 113.776244 | 0.0994 | 28876.920802 |
Index 20
ARIMAX(0, 1, 1) : Prediction: 2021-11-14 [462.44274815]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-11-14 | 540745.2 | 2.9 | 81.316586 | 114.134864 | 0.083 | 29773.53907 |
Index 21
ARIMAX(0, 1, 1) : Prediction: 2021-11-21 [458.89018475]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-11-21 | 540745.2 | 2.9 | 81.962652 | 113.392382 | 0.0684 | 29309.477076 |
Index 22
ARIMAX(0, 1, 1) : Prediction: 2021-11-28 [441.77058407]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-11-28 | 540745.2 | 2.9 | 78.321511 | 114.493276 | 0.0802 | 29801.798751 |
Index 23
ARIMAX(0, 1, 1) : Prediction: 2021-12-05 [434.39410034]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-12-05 | 540745.2 | 2.9 | 77.520128 | 114.762155 | 0.08475 | 29272.477744 |
print("Rolling ARIMAX3 / all exog")
make_plot(y_train, y_test, pred_rolling_arimax_all3, "Rolling ARIMAX3 / all exog")
metrics_rolling_arimax_all3 = calc_metrics(y_test, pred_rolling_arimax_all3, return_dict=True)
Rolling ARIMAX3 / all exog
Test Error Metrics MAE: 8.0231 MSE: 123.2682 MAPE: 0.0179 sMAPE: 0.0178 MASE: 10.6373
pred_rolling_arimax_n2253 = rolling_arimax3(exog_list=['n225'], verbose=False)
print("Rolling ARIMAX3 / n225")
make_plot(y_train, y_test, pred_rolling_arimax_n2253, "Rolling ARIMAX3 / n225")
metrics_rolling_arimax_n2253 = calc_metrics(y_test, pred_rolling_arimax_n2253, return_dict=True)
Rolling ARIMAX3 / n225
Test Error Metrics MAE: 8.0718 MSE: 124.1081 MAPE: 0.0181 sMAPE: 0.0179 MASE: 10.7018
pred_rolling_arimax_oilusdjpy3 = rolling_arimax3(exog_list=['oil', 'usdjpy'], verbose=False)
print("Rolling ARIMAX3 / Oil & USDJPY")
make_plot(y_train, y_test, pred_rolling_arimax_oilusdjpy3, "Rolling ARIMAX3 / Oil & USDJPY")
metrics_rolling_arimax_oilusdjpy3 = calc_metrics(y_test, pred_rolling_arimax_oilusdjpy3, return_dict=True)
Rolling ARIMAX3 / Oil & USDJPY
Test Error Metrics MAE: 7.8984 MSE: 122.567 MAPE: 0.0177 sMAPE: 0.0175 MASE: 10.472
pred_rolling_arimax_gdpunemp3 = rolling_arimax3(exog_list=['jp_gdp', 'jp_unemp'], verbose=False)
print("Rolling ARIMAX3 / GDP & Unemp")
make_plot(y_train, y_test, pred_rolling_arimax_gdpunemp3, "Rolling ARIMAX3 / GDP & Unemp")
metrics_rolling_arimax_gdpunemp3 = calc_metrics(y_test, pred_rolling_arimax_gdpunemp3, return_dict=True)
Rolling ARIMAX3 / GDP & Unemp
Test Error Metrics MAE: 7.6359 MSE: 118.8654 MAPE: 0.0171 sMAPE: 0.0169 MASE: 10.1239
# Function to run rolling fit
def rolling_arimax4(exog_list, train_df=df_train, test_df=df_test, verbose=False):
"""
***** Rolling ARIMAX(3) uses whole training data, while Rolling ARIMAX(4) uses 52 latest values for prediction *****
Input:
(1) exog_list: column names for external regressors
(2) train_df: df_train
(3) test_df: df for extracting index / exog for training
Process:
(1) Build model using auto_arima() and whole training data (eneos + external regressors / use only 52 latest values)
(2) Make one-step ahead prediction for external regressors using actual values available at the point of prediction
(3) Use model from (1) and external regressors from (2) to make one-step ahead prediction for eneos
(4) Store predicted eneos and predicted external regressors as training data
(5) Repeat (1) through (4) over test data periods
Outoput:
- Prediction for Eneos (23-step ahead)
"""
# Extract eneos from train/test df as train_y and test_y (use last 52 values for train_y)
train_y = train_df.copy()[-52:]["eneos"]
test_y = test_df.copy()["eneos"]
# Create df for external regressors (Use last 52 values)
df_X = train_df.copy()[-52:][exog_list]
# Series to store predicted values
y_pred = pd.Series(name="eneos")
for i in range(test_df.shape[0]):
# Create ARIMAX model using training eneos and external regressors (df_X)
arimax = auto_arima(train_y, X=df_X, start_p=0, d=1, start_q=0, seasonal=False)
arimax_order = arimax.get_params()["order"]
# Create df for predicted external regressors
df_pred_exog = pd.DataFrame(columns=exog_list, index=[test_df.index[i]])
# Make one-step ahead forecast for external regressors
for c in exog_list:
arima_exog = auto_arima(df_X[c], start_p=0, d=1, start_q=0, seasonal=False)
pred_exog = arima_exog.predict(n_periods=1)
df_pred_exog[c] = pred_exog
# Make prediction
ypred = arimax.predict(n_periods=1, X=df_pred_exog)
ypred = pd.Series(ypred, name="eneos", index=[test_df.index[i]])
# Store predicted value into y_pred
y_pred = y_pred.append(ypred)
# Add 1-step ahead true observation to train_y and drop first observation from train_y
train_y = train_y.append(test_y[i:i+1])[1:]
# Add 1-step ahead true external regressors into df_X and drop first observation from df_X
df_X = df_X.append(test_df.copy().iloc[i][exog_list])[1:]
# Print progress
if verbose:
print(f"Index {i + 1}")
print(f" ARIMAX{str(arimax_order)} : Prediction: {ypred.index[0].strftime('%Y-%m-%d')} {ypred.values}")
print(" One-step ahead forecast for External Regressors:")
display(df_pred_exog)
return y_pred
pred_rolling_arimax_all4 = rolling_arimax4(exog_list=['jp_gdp', 'jp_unemp', 'oil', 'usdjpy', 'jp_i', 'n225'], verbose=True)
Index 1
ARIMAX(0, 1, 0) : Prediction: 2021-07-04 [468.85475778]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-07-04 | 544693.4 | 3.1 | 74.143539 | 110.632399 | 0.0604 | 28871.11152 |
Index 2
ARIMAX(0, 1, 0) : Prediction: 2021-07-11 [470.37104768]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-07-11 | 538923.3 | 2.9 | 74.727333 | 110.9312 | 0.0608 | 28952.238833 |
Index 3
ARIMAX(0, 1, 0) : Prediction: 2021-07-18 [470.00972894]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-07-18 | 538923.3 | 2.9 | 73.987647 | 110.617801 | 0.033 | 28442.70396 |
Index 4
ARIMAX(0, 1, 0) : Prediction: 2021-07-25 [474.63855928]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-07-25 | 538923.3 | 2.9 | 73.81251 | 110.179399 | 0.0206 | 28546.390791 |
Index 5
ARIMAX(0, 1, 0) : Prediction: 2021-08-01 [455.57135168]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-01 | 538923.3 | 2.9 | 70.252588 | 109.944598 | 0.015667 | 27630.922079 |
Index 6
ARIMAX(1, 1, 2) : Prediction: 2021-08-08 [463.54081244]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-08 | 538923.3 | 2.9 | 73.361412 | 109.990399 | 0.019 | 27793.84064 |
Index 7
ARIMAX(0, 1, 1) : Prediction: 2021-08-15 [469.58547707]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-15 | 538923.3 | 2.9 | 70.037725 | 109.449399 | 0.014 | 27711.017578 |
Index 8
ARIMAX(1, 1, 0) : Prediction: 2021-08-22 [468.58825959]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-22 | 538923.3 | 2.9 | 68.836118 | 110.388 | 0.03225 | 28085.272924 |
Index 9
ARIMAX(1, 1, 2) : Prediction: 2021-08-29 [421.50703324]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-08-29 | 538923.3 | 2.9 | 65.481647 | 109.6136 | 0.0158 | 27365.598047 |
Index 10
ARIMAX(1, 1, 2) : Prediction: 2021-09-05 [418.42460147]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-09-05 | 538923.3 | 2.9 | 68.094863 | 109.854399 | 0.0258 | 27666.914062 |
Index 11
ARIMAX(1, 1, 2) : Prediction: 2021-09-12 [434.93178222]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-09-12 | 538923.3 | 2.9 | 69.773039 | 109.9812 | 0.0352 | 28502.108601 |
Index 12
ARIMAX(0, 1, 3) : Prediction: 2021-09-19 [453.68096809]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-09-19 | 538923.3 | 2.9 | 69.559529 | 109.9758 | 0.0496 | 30158.77932 |
Index 13
ARIMAX(1, 1, 2) : Prediction: 2021-09-26 [464.4047016]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-09-26 | 538923.3 | 2.9 | 72.313647 | 109.742001 | 0.0504 | 30633.210876 |
Index 14
ARIMAX(1, 1, 2) : Prediction: 2021-10-03 [467.00260623]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-03 | 538923.3 | 2.9 | 72.830745 | 109.740999 | 0.051333 | 30038.579306 |
Index 15
ARIMAX(1, 1, 2) : Prediction: 2021-10-10 [455.94011783]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-10 | 540745.2 | 2.8 | 76.14549 | 111.330598 | 0.0744 | 29759.022281 |
Index 16
ARIMAX(0, 1, 3) : Prediction: 2021-10-17 [456.90419329]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-17 | 540745.2 | 2.8 | 79.25251 | 111.295401 | 0.072 | 27904.605859 |
Index 17
ARIMAX(0, 1, 3) : Prediction: 2021-10-24 [463.24581137]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-24 | 540745.2 | 2.8 | 81.97502 | 113.420599 | 0.0874 | 28497.729687 |
Index 18
ARIMAX(0, 1, 1) : Prediction: 2021-10-31 [469.26151881]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-10-31 | 540745.2 | 2.8 | 84.381098 | 114.468247 | 0.0928 | 29001.992188 |
Index 19
ARIMAX(0, 1, 1) : Prediction: 2021-11-07 [466.08425237]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-11-07 | 540745.2 | 2.8 | 84.748627 | 113.986573 | 0.0994 | 28886.404453 |
Index 20
ARIMAX(0, 1, 1) : Prediction: 2021-11-14 [463.90703547]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-11-14 | 540745.2 | 2.9 | 82.594471 | 114.220364 | 0.083 | 29643.47998 |
Index 21
ARIMAX(0, 1, 1) : Prediction: 2021-11-21 [460.80152421]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-11-21 | 540745.2 | 2.9 | 82.722431 | 113.700043 | 0.0684 | 29357.424219 |
Index 22
ARIMAX(0, 1, 1) : Prediction: 2021-11-28 [442.14873412]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-11-28 | 540745.2 | 2.9 | 79.670275 | 114.493848 | 0.0802 | 29723.555859 |
Index 23
ARIMAX(1, 1, 2) : Prediction: 2021-12-05 [444.98068675]
One-step ahead forecast for External Regressors:
| jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|
| 2021-12-05 | 540745.2 | 2.9 | 78.429085 | 115.119297 | 0.08475 | 29331.916992 |
print("Rolling ARIMAX4 / all exog")
make_plot(y_train, y_test, pred_rolling_arimax_all4, "Rolling ARIMAX4 / all exog")
metrics_rolling_arimax_all4 = calc_metrics(y_test, pred_rolling_arimax_all4, return_dict=True)
Rolling ARIMAX4 / all exog
Test Error Metrics MAE: 7.7303 MSE: 131.0375 MAPE: 0.0174 sMAPE: 0.0171 MASE: 10.2491
pred_rolling_arimax_n2254 = rolling_arimax4(exog_list=['n225'], verbose=False)
print("Rolling ARIMAX4 / n225")
make_plot(y_train, y_test, pred_rolling_arimax_n2254, "Rolling ARIMAX4 / n225")
metrics_rolling_arimax_n2254 = calc_metrics(y_test, pred_rolling_arimax_n2254, return_dict=True)
Rolling ARIMAX4 / n225
Test Error Metrics MAE: 7.411 MSE: 127.3857 MAPE: 0.0166 sMAPE: 0.0164 MASE: 9.8258
pred_rolling_arimax_oilusdjpy4 = rolling_arimax4(exog_list=['oil', 'usdjpy'], verbose=False)
print("Rolling ARIMAX4 / Oil & USDJPY")
make_plot(y_train, y_test, pred_rolling_arimax_oilusdjpy4, "Rolling ARIMAX3 / Oil & USDJPY")
metrics_rolling_arimax_oilusdjpy4 = calc_metrics(y_test, pred_rolling_arimax_oilusdjpy4, return_dict=True)
Rolling ARIMAX4 / Oil & USDJPY
Test Error Metrics MAE: 8.2142 MSE: 127.4794 MAPE: 0.0185 sMAPE: 0.0182 MASE: 10.8906
pred_rolling_arimax_gdpunemp4 = rolling_arimax4(exog_list=['jp_gdp', 'jp_unemp'], verbose=False)
print("Rolling ARIMAX4 / GDP & Unemp")
make_plot(y_train, y_test, pred_rolling_arimax_gdpunemp4, "Rolling ARIMAX4 / GDP & Unemp")
metrics_rolling_arimax_gdpunemp4 = calc_metrics(y_test, pred_rolling_arimax_gdpunemp4, return_dict=True)
Rolling ARIMAX4 / GDP & Unemp
Test Error Metrics MAE: 8.084 MSE: 123.1674 MAPE: 0.0181 sMAPE: 0.0179 MASE: 10.718
# Library
from statsmodels.tsa.vector_ar.var_model import VAR
# Function to run VAR
def rolling_var(exog_list, train_diff_raw=df_train_diff, test_diff_raw=df_test_diff, train_raw=df_train, verbose=False):
"""
Input:
(1) exog_list: column names for external regressors
(2) train_diff_raw: df_train_diff (differenced training data)
(3) test_diff_raw: df for extracting index / exog for training
(4) train_raw: df_train (non-differenced training data)
Process:
(1) Build model using VAR() with optimal order p in terms of AIC using whole differenced training data
(2) Make one-step ahead prediction & Store predicted values
(3) Add 1-step ahead true values into differenced training data and drop first observation from it
(5) Repeat (1) through (3) over test data periods
Outoput:
- Prediction for Eneos (23-step ahead)
"""
# Slice data by eneos and specified exog
train_diff = train_diff_raw[["eneos"] + exog_list]
test_diff = test_diff_raw[["eneos"] + exog_list]
train = train_raw[["eneos"] + exog_list]
# DF to store predicted values
df_pred = pd.DataFrame(columns=train.columns)
# Fit var model and make 1-step ahead prediction
for i in range(test_diff.shape[0]):
# Create instance
var_model=VAR(train_diff)
# Run VAR with different order (1-20)
results_aic =[]
for p in range(1,21):
results = var_model.fit(p)
results_aic.append(results.aic)
# Store optimal p order
optimal_p = list(np.arange(1,21,1))[np.argmin(results_aic)]
# Refit var with optimal p
best_var = var_model.fit(optimal_p)
# Make 1-step ahead forecast
yhat_var = best_var.forecast(var_model.endog, steps=1)
# clean up yhat
df_yhat_var = pd.DataFrame(yhat_var)
df_yhat_var.index = [test_diff.index[i]]
df_yhat_var.columns = test_diff.columns
# Store df_yhat_var
df_pred = df_pred.append(df_yhat_var)
# Add actual 1-steap ahead actual test data into training set and drop 1st observation from training data
train_diff = train_diff.append(test_diff.copy().iloc[i])[1:]
# Print progress
if verbose:
print(f"Index {i + 1}")
print("Optimal p with minimum AIC: ", optimal_p)
print("One-step ahead forecast (differened data)")
display(df_yhat_var)
# Recover first order differencing
df_pred_var = pd.concat([train.tail(1), df_pred], axis=0).cumsum()
df_pred_var = df_pred_var[-23:]
# Extract predictions for eneos
y_pred_var = df_pred_var["eneos"]
# Return prediction
return y_pred_var
pred_rolling_var_all = rolling_var(exog_list=['jp_gdp', 'jp_unemp', 'oil', 'usdjpy', 'jp_i', 'n225'], verbose=True)
Index 1 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-07-04 | -1.513281 | 2256.425159 | -0.007604 | -0.876597 | 0.122273 | -0.006571 | -58.545145 |
Index 2 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-07-11 | 0.790945 | 379.409512 | -0.007532 | -0.236408 | 0.088532 | -0.004658 | 130.51646 |
Index 3 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-07-18 | 1.780447 | -62.975103 | 0.008574 | -1.289651 | 0.061688 | 0.001161 | 39.305007 |
Index 4 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-07-25 | 2.142438 | 105.790606 | 0.005648 | -1.660954 | -0.272097 | -0.000332 | 58.499436 |
Index 5 Optimal p with minimum AIC: 4 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-01 | -9.598102 | 3509.557975 | 0.052101 | -1.14103 | 0.540356 | 0.004722 | -246.397448 |
Index 6 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-08 | 5.806878 | -407.818371 | -0.014193 | 0.187219 | -0.041772 | 0.002207 | 140.619358 |
Index 7 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-15 | -4.06812 | -2443.752324 | 0.021735 | -0.613647 | 0.051894 | 0.001095 | -165.265719 |
Index 8 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-22 | 5.779343 | 806.516374 | -0.000558 | 0.578725 | 0.151531 | 0.003868 | 207.148541 |
Index 9 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-29 | -14.277217 | -1526.793746 | 0.020112 | -0.899097 | -0.540703 | -0.011921 | -287.263675 |
Index 10 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-09-05 | 3.015781 | 763.635324 | -0.015081 | 0.373261 | -0.00101 | 0.005832 | 200.867401 |
Index 11 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-09-12 | 3.615822 | -1275.107669 | 0.009644 | 0.98505 | 0.260531 | -0.002706 | 196.081054 |
Index 12 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-09-19 | 5.797649 | -978.249732 | 0.006768 | 2.310025 | -0.112974 | 0.007109 | 371.706652 |
Index 13 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-09-26 | 2.621508 | 2832.542566 | -0.016965 | 2.689276 | 0.049251 | 0.001346 | 244.144054 |
Index 14 Optimal p with minimum AIC: 4 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-03 | -8.355479 | 1962.103571 | -0.00049 | 0.722212 | 0.023323 | -0.006698 | -39.499036 |
Index 15 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-10 | 2.631177 | 797.496434 | -0.023534 | -0.482489 | 0.103853 | -0.005151 | 130.717288 |
Index 16 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-17 | -0.752958 | -286.339306 | -0.027094 | -2.232542 | 0.50604 | -0.006194 | -239.274927 |
Index 17 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-24 | 13.361011 | -2983.830347 | -0.006674 | 0.011305 | 0.244386 | 0.003543 | 374.862648 |
Index 18 Optimal p with minimum AIC: 4 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-31 | 1.484041 | -263.892506 | 0.003337 | -1.706374 | 0.071836 | 0.008001 | 155.684278 |
Index 19 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-11-07 | -2.50428 | 1170.709418 | -0.020447 | 0.091229 | 0.191562 | 0.008108 | 21.573239 |
Index 20 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-11-14 | 4.948095 | -748.361407 | 0.013751 | 1.234919 | -0.052447 | 0.001858 | 277.452321 |
Index 21 Optimal p with minimum AIC: 1 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-11-21 | -2.789755 | 344.983169 | 0.002458 | -0.048189 | -0.063408 | 0.000351 | -43.117886 |
Index 22 Optimal p with minimum AIC: 1 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-11-28 | -1.484458 | -1067.674005 | 0.012255 | -0.120835 | -0.115756 | -0.005088 | 66.204575 |
Index 23 Optimal p with minimum AIC: 3 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-12-05 | 0.344676 | 76.697144 | -0.004458 | -0.050559 | 0.017206 | -0.00946 | 18.433166 |
print("Rolling Var / all exog")
make_plot(y_train, y_test, pred_rolling_var_all, "Rolling Var / all exog")
metrics_rolling_var_all = calc_metrics(y_test, pred_rolling_var_all, return_dict=True)
Rolling Var / all exog
Test Error Metrics MAE: 14.9596 MSE: 422.0461 MAPE: 0.0339 sMAPE: 0.0328 MASE: 19.834
pred_rolling_var_n225 = rolling_var(exog_list=['n225'])
print("Rolling Var / n225")
make_plot(y_train, y_test, pred_rolling_var_n225, "Rolling Var / n225")
metrics_rolling_var_n225 = calc_metrics(y_test, pred_rolling_var_n225, return_dict=True)
Rolling Var / n225
Test Error Metrics MAE: 9.6287 MSE: 170.5378 MAPE: 0.0218 sMAPE: 0.0214 MASE: 12.766
pred_rolling_var_oilusdjpy = rolling_var(exog_list=['oil', 'usdjpy'])
print("Rolling Var / oil & usdjpy")
make_plot(y_train, y_test, pred_rolling_var_oilusdjpy, "Rolling Var / oil & usdjpy")
metrics_rolling_var_oilusdjpy = calc_metrics(y_test, pred_rolling_var_oilusdjpy, return_dict=True)
Rolling Var / oil & usdjpy
Test Error Metrics MAE: 12.0597 MSE: 294.9881 MAPE: 0.0274 sMAPE: 0.0267 MASE: 15.9891
pred_rolling_var_gdpunemp = rolling_var(exog_list=['jp_gdp', 'jp_unemp'])
print("Rolling Var / GDP & Unemp")
make_plot(y_train, y_test, pred_rolling_var_gdpunemp, "Rolling Var / GDP & Unemp")
metrics_rolling_var_gdpunemp = calc_metrics(y_test, pred_rolling_var_gdpunemp, return_dict=True)
Rolling Var / GDP & Unemp
Test Error Metrics MAE: 9.5726 MSE: 175.8181 MAPE: 0.0217 sMAPE: 0.0213 MASE: 12.6917
# Library
from statsmodels.tsa.vector_ar.var_model import VAR
# Function to run VAR
def rolling_var2(exog_list, train_diff_raw=df_train_diff, test_diff_raw=df_test_diff, train_raw=df_train, verbose=False):
"""
***** Use only last 52 observations as training instead of whole training data *****
Input:
(1) exog_list: column names for external regressors
(2) train_diff_raw: df_train_diff (differenced training data)
(3) test_diff_raw: df for extracting index / exog for training
(4) train_raw: df_train (non-differenced training data)
Process:
(1) Build model using VAR() with optimal order p in terms of AIC using whole differenced training data
(2) Make one-step ahead prediction & Store predicted values
(3) Add 1-step ahead true values into differenced training data and drop first observation from it
(5) Repeat (1) through (3) over test data periods
Outoput:
- Prediction for Eneos (23-step ahead)
"""
# Slice data by eneos and specified exog
train_diff = train_diff_raw[["eneos"] + exog_list][-52:]
test_diff = test_diff_raw[["eneos"] + exog_list]
train = train_raw[["eneos"] + exog_list][-52:]
# DF to store predicted values
df_pred = pd.DataFrame(columns=train.columns)
# Fit var model and make 1-step ahead prediction
for i in range(test_diff.shape[0]):
# Create instance
var_model=VAR(train_diff)
# Run VAR with different order (1-20)
results_aic =[]
for p in range(1,6):
results = var_model.fit(p)
results_aic.append(results.aic)
# Store optimal p order
optimal_p = list(np.arange(1,6,1))[np.argmin(results_aic)]
# Refit var with optimal p
best_var = var_model.fit(optimal_p)
# Make 1-step ahead forecast
yhat_var = best_var.forecast(var_model.endog, steps=1)
# clean up yhat
df_yhat_var = pd.DataFrame(yhat_var)
df_yhat_var.index = [test_diff.index[i]]
df_yhat_var.columns = test_diff.columns
# Store df_yhat_var
df_pred = df_pred.append(df_yhat_var)
# Add actual 1-steap ahead actual test data into training set and drop 1st observation from training data
train_diff = train_diff.append(test_diff.copy().iloc[i])[1:]
# Print progress
if verbose:
print(f"Index {i + 1}")
print("Optimal p with minimum AIC: ", optimal_p)
print("One-step ahead forecast (differened data)")
display(df_yhat_var)
# Recover first order differencing
df_pred_var = pd.concat([train.tail(1), df_pred], axis=0).cumsum()
df_pred_var = df_pred_var[-23:]
# Extract predictions for eneos
y_pred_var = df_pred_var["eneos"]
# Return prediction
return y_pred_var
pred_rolling_var_all2 = rolling_var2(exog_list=['jp_gdp', 'jp_unemp', 'oil', 'usdjpy', 'jp_i', 'n225'], verbose=True)
Index 1 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-07-04 | -8.314044 | 1032.870871 | -0.024384 | -0.54334 | 0.36973 | -0.016561 | -197.944102 |
Index 2 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-07-11 | 13.773715 | 767.367637 | 0.166496 | 6.48703 | -0.488471 | -0.001586 | 1484.56999 |
Index 3 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-07-18 | 34.022276 | -2003.867009 | 0.051157 | 3.874497 | 0.180855 | -0.019495 | 692.113332 |
Index 4 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-07-25 | 13.704819 | 1232.097573 | -0.011492 | -4.396359 | -0.035721 | -0.025132 | -243.306171 |
Index 5 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-01 | -9.979757 | 2676.471064 | 0.130756 | 0.105056 | 0.668701 | 0.000527 | 460.34723 |
Index 6 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-08 | -13.51212 | 901.707149 | 0.051966 | -1.730317 | 1.13102 | -0.001993 | 43.375754 |
Index 7 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-15 | -13.727547 | 4590.046158 | 0.12691 | -1.019986 | -1.050976 | 0.020383 | -360.832896 |
Index 8 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-22 | 3.81321 | -2514.132143 | -0.057127 | 2.489032 | -2.343505 | -0.015342 | -199.579536 |
Index 9 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-08-29 | -24.016602 | 6414.83794 | 0.086795 | -0.739135 | 0.482047 | -0.068441 | -512.76186 |
Index 10 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-09-05 | 17.115559 | 537.568207 | 0.006975 | 7.979582 | 0.349821 | 0.005611 | 2274.662849 |
Index 11 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-09-12 | -10.198847 | -2075.240077 | -0.048603 | -2.083864 | -0.662296 | 0.017262 | -575.768676 |
Index 12 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-09-19 | 9.770602 | 2109.146167 | 0.045889 | 1.082521 | 0.516407 | 0.020698 | 600.517591 |
Index 13 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-09-26 | -13.924356 | -121.961974 | -0.035423 | 2.282769 | -0.232916 | -0.007618 | -1345.55989 |
Index 14 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-03 | 34.107402 | -2466.385345 | -0.140396 | 4.307211 | 1.001937 | 0.010731 | 977.954973 |
Index 15 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-10 | 10.576827 | -4474.261287 | -0.168603 | 1.991815 | 1.174309 | -0.010111 | 1361.117648 |
Index 16 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-17 | 2.898566 | -3359.257614 | -0.108837 | -3.612282 | 1.579853 | -0.034517 | -315.72693 |
Index 17 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-24 | 18.073238 | 1446.265138 | 0.009273 | -2.180989 | 0.572311 | 0.000718 | -59.531563 |
Index 18 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-10-31 | -16.561538 | -1990.81454 | 0.101579 | -7.626686 | 1.106887 | -0.00664 | 161.39252 |
Index 19 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-11-07 | 29.607473 | -2719.392479 | -0.115025 | 3.197274 | 1.370634 | 0.011837 | 1120.007327 |
Index 20 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-11-14 | 16.558024 | 482.097909 | 0.004681 | 6.364389 | 1.439378 | 0.048759 | 2422.219235 |
Index 21 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-11-21 | -16.262707 | -684.00595 | -0.006043 | -3.499908 | -1.380322 | -0.03227 | -1301.186927 |
Index 22 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-11-28 | 3.328882 | -370.697842 | -0.140702 | 0.426874 | -0.57935 | -0.020654 | -1030.737425 |
Index 23 Optimal p with minimum AIC: 5 One-step ahead forecast (differened data)
| eneos | jp_gdp | jp_unemp | oil | usdjpy | jp_i | n225 | |
|---|---|---|---|---|---|---|---|
| 2021-12-05 | -21.840951 | -2089.261937 | -0.106138 | 1.60291 | -0.987807 | -0.023134 | -63.743922 |
print("Rolling Var2 / all exog")
make_plot(y_train, y_test, pred_rolling_var_all2, "Rolling Var2 / all exog")
metrics_rolling_var_all2 = calc_metrics(y_test, pred_rolling_var_all2, return_dict=True)
Rolling Var2 / all exog
Test Error Metrics MAE: 50.3905 MSE: 3522.3065 MAPE: 0.1121 sMAPE: 0.1041 MASE: 66.8095
pred_rolling_var_n2252 = rolling_var2(exog_list=['n225'])
print("Rolling Var2 / n225")
make_plot(y_train, y_test, pred_rolling_var_n2252, "Rolling Var2 / n225")
metrics_rolling_var_n2252 = calc_metrics(y_test, pred_rolling_var_n2252, return_dict=True)
Rolling Var2 / n225
Test Error Metrics MAE: 18.0353 MSE: 531.7428 MAPE: 0.0408 sMAPE: 0.0394 MASE: 23.9118
pred_rolling_var_oilusdjpy2 = rolling_var2(exog_list=['oil', 'usdjpy'])
print("Rolling Var2 / oil & usdjpy")
make_plot(y_train, y_test, pred_rolling_var_oilusdjpy2, "Rolling Var2 / oil & usdjpy")
metrics_rolling_var_oilusdjpy2 = calc_metrics(y_test, pred_rolling_var_oilusdjpy2, return_dict=True)
Rolling Var2 / oil & usdjpy
Test Error Metrics MAE: 14.6472 MSE: 353.8647 MAPE: 0.0331 sMAPE: 0.0322 MASE: 19.4197
pred_rolling_var_gdpunemp2 = rolling_var2(exog_list=['jp_gdp', 'jp_unemp'])
print("Rolling Var2 / GDP & Unemp")
make_plot(y_train, y_test, pred_rolling_var_gdpunemp2, "Rolling Var2 / GDP & Unemp")
metrics_rolling_var_gdpunemp2 = calc_metrics(y_test, pred_rolling_var_gdpunemp2, return_dict=True)
Rolling Var2 / GDP & Unemp
Test Error Metrics MAE: 28.0494 MSE: 1056.3627 MAPE: 0.0629 sMAPE: 0.0603 MASE: 37.1889
Roplling Univariate ARIMA (4), Rolling ARIMAX with External Regressors (4), and Rolling VAR# Store all predictions and metrics
weekly_rolling_pred = dict()
weekly_rolling_metrics = dict()
# Rolling Univariate ARIMA(4)
weekly_rolling_pred["Rolling ARIMA / No-exog"] = pred_airma_rolling_uni4
weekly_rolling_metrics["Rolling ARIMA / No-exog"] = metrics_airma_rolling_uni4
# Rolling ARIMAX with external regressors (4)
weekly_rolling_pred["Rolling ARIMAX / All"] = pred_rolling_arimax_all4
weekly_rolling_metrics["Rolling ARIMAX / All"] = metrics_rolling_arimax_all4
weekly_rolling_pred["Rolling ARIMAX / Nikkei225"] = pred_rolling_arimax_n2254
weekly_rolling_metrics["Rolling ARIMAX / Nikkei225"] = metrics_rolling_arimax_n2254
weekly_rolling_pred["Rolling ARIMAX / Oil & USDJPY"] = pred_rolling_arimax_oilusdjpy4
weekly_rolling_metrics["Rolling ARIMAX / Oil & USDJPY"] = metrics_rolling_arimax_oilusdjpy4
weekly_rolling_pred["Rolling ARIMAX / GDP & Unemp"] = pred_rolling_arimax_gdpunemp4
weekly_rolling_metrics["Rolling ARIMAX / GDP & Unemp"] = metrics_rolling_arimax_gdpunemp4
# Rolling VAR
weekly_rolling_pred["Rolling VAR / All"] = pred_rolling_var_all
weekly_rolling_metrics["Rolling VAR / All"] = metrics_rolling_var_all
weekly_rolling_pred["Rolling VAR / Nikkei225"] = pred_rolling_var_n225
weekly_rolling_metrics["Rolling VAR / Nikkei225"] = metrics_rolling_var_n225
weekly_rolling_pred["Rolling VAR / Oil & USDJPY"] = pred_rolling_var_oilusdjpy
weekly_rolling_metrics["Rolling VAR / Oil & USDJPY"] = metrics_rolling_var_oilusdjpy
weekly_rolling_pred["Rolling VAR / GDP & Unemp"] = pred_rolling_var_gdpunemp
weekly_rolling_metrics["Rolling VAR / GDP & Unemp"] = metrics_rolling_var_gdpunemp
pd.DataFrame(weekly_rolling_metrics).T
| MAE | MSE | MAPE | sMAPE | MASE | |
|---|---|---|---|---|---|
| Rolling ARIMA / No-exog | 8.2791 | 127.8672 | 0.0185 | 0.0184 | 10.9768 |
| Rolling ARIMAX / All | 7.7303 | 131.0375 | 0.0174 | 0.0171 | 10.2491 |
| Rolling ARIMAX / Nikkei225 | 7.4110 | 127.3857 | 0.0166 | 0.0164 | 9.8258 |
| Rolling ARIMAX / Oil & USDJPY | 8.2142 | 127.4794 | 0.0185 | 0.0182 | 10.8906 |
| Rolling ARIMAX / GDP & Unemp | 8.0840 | 123.1674 | 0.0181 | 0.0179 | 10.7180 |
| Rolling VAR / All | 14.9596 | 422.0461 | 0.0339 | 0.0328 | 19.8340 |
| Rolling VAR / Nikkei225 | 9.6287 | 170.5378 | 0.0218 | 0.0214 | 12.7660 |
| Rolling VAR / Oil & USDJPY | 12.0597 | 294.9881 | 0.0274 | 0.0267 | 15.9891 |
| Rolling VAR / GDP & Unemp | 9.5726 | 175.8181 | 0.0217 | 0.0213 | 12.6917 |
pd.DataFrame(weekly_rolling_metrics).T.sort_values("sMAPE")
| MAE | MSE | MAPE | sMAPE | MASE | |
|---|---|---|---|---|---|
| Rolling ARIMAX / Nikkei225 | 7.4110 | 127.3857 | 0.0166 | 0.0164 | 9.8258 |
| Rolling ARIMAX / All | 7.7303 | 131.0375 | 0.0174 | 0.0171 | 10.2491 |
| Rolling ARIMAX / GDP & Unemp | 8.0840 | 123.1674 | 0.0181 | 0.0179 | 10.7180 |
| Rolling ARIMAX / Oil & USDJPY | 8.2142 | 127.4794 | 0.0185 | 0.0182 | 10.8906 |
| Rolling ARIMA / No-exog | 8.2791 | 127.8672 | 0.0185 | 0.0184 | 10.9768 |
| Rolling VAR / GDP & Unemp | 9.5726 | 175.8181 | 0.0217 | 0.0213 | 12.6917 |
| Rolling VAR / Nikkei225 | 9.6287 | 170.5378 | 0.0218 | 0.0214 | 12.7660 |
| Rolling VAR / Oil & USDJPY | 12.0597 | 294.9881 | 0.0274 | 0.0267 | 15.9891 |
| Rolling VAR / All | 14.9596 | 422.0461 | 0.0339 | 0.0328 | 19.8340 |
fig, ax = plt.subplots(figsize=(16,5))
y_train[-20:].plot(ax=ax, label="Training")
y_test.plot(ax=ax, label="Test", color="black")
for model in weekly_rolling_pred.keys():
weekly_rolling_pred[model].plot(ax=ax, label=model, linestyle="dashed")
plt.legend()
plt.grid()
plt.show()
So far, we built models using ARIMA, VAR, and Prophet to predict "TOYKO GAS" stock price (weekly average for daily close price) in latter half of 2021 (training data is from 2017-01 to 2021-06). Also, we used the following as external regressors.
Based on sMAPE criteria, VAR(1) using TOYKO GAS price and Nikkei 225 index performs best for test data. Also, other VAR models seem to perform better than ARIMA and Prophet.